Sep 05 02:17:36.248017 ip-10-0-140-13 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:36.636678 ip-10-0-140-13 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.636678 ip-10-0-140-13 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:36.636678 ip-10-0-140-13 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.636678 ip-10-0-140-13 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:36.636678 ip-10-0-140-13 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.639182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.639106    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645346    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645362    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645366    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645369    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645372    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.645367 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645374    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645378    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645381    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645384    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645387    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645389    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645392    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645395    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645397    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645400    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645402    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645405    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645408    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645412    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645417    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645420    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645427    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645430    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645433    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645437    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.645628 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645440    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645443    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645446    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645449    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645452    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645456    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645458    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645461    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645463    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645466    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645468    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645471    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645473    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645476    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645479    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645482    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645485    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645488    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645490    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.646136 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645493    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645495    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645498    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645500    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645503    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645505    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645509    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645511    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645514    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645517    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645519    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645522    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645524    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645527    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645529    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645532    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645534    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645537    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645540    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645542    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.646618 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645545    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645547    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645549    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645570    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645573    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645577    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645582    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645586    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645589    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645592    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645595    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645599    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645601    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645604    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645607    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645610    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645613    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645615    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645618    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645621    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.647134 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645624    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645627    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645630    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645632    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645635    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645638    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645641    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645643    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645646    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645648    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645652    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645654    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645657    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645659    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.647622 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.645662    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647954    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647963    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647966    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647969    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647972    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647975    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647978    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.647978 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647981    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647984    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647987    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647990    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647993    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647995    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.647998    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648001    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648003    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648006    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648009    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648012    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648014    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648017    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648019    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648022    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648026    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648030    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648032    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648036    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.648181 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648039    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648041    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648044    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648047    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648049    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648052    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648054    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648057    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648059    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648062    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648064    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648067    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648069    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648072    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648075    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648078    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648081    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648083    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648086    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648088    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.648833 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648091    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648094    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648096    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648099    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648102    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648104    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648107    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648110    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648112    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648115    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648117    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648120    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648122    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648125    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648127    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648131    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648135    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648137    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648140    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.649352 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648143    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648145    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648148    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648151    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648153    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648156    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648159    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648161    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648166    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648170    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648172    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648175    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648178    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648181    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648184    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648186    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648189    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648192    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648195    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.649843 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648198    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648201    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648203    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648206    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648209    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648211    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648214    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648216    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648219    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648222    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648225    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648227    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648230    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648233    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648589    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648594    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648598    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648600    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648603    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648606    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.650310 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648609    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648611    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648614    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648616    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648619    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648622    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648625    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648627    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648630    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648632    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648635    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648638    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648640    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648643    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648645    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648648    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648651    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648653    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648656    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648658    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.650818 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648660    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648664    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648667    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648669    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648672    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648674    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648677    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648679    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648684    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648687    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648691    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648694    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648697    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648700    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648703    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648705    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648708    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648711    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648714    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.651312 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648718    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648721    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648724    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648727    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648730    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648732    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648735    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648738    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648757    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648760    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648762    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648765    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648767    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648770    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648773    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648776    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648779    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648782    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648784    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648787    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.651793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648790    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648792    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648795    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648797    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648800    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648803    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648805    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648808    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648810    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648813    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648816    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648818    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648823    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648827    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648829    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648832    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648835    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648839    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648841    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648844    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.652287 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648847    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648850    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648852    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648855    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648858    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648860    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648863    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648866    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648868    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648871    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648874    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648876    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648879    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.648881    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.648952    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.648985    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.648990    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.648994    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.648998    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649001    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649005    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:36.652802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649009    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649013    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649016    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649019    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649022    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649025    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649028    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649031    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649034    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649037    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649041    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649045    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649048    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649051    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649054    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649058    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649061    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649064    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649070    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649073    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649076    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649079    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649082    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649085    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:36.653309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649088    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649092    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649095    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649098    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649101    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649104    2560 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649107    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649110    2560 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649113    2560 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649116    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649119    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649122    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649125    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649128    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649131    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649134    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649137    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649140    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649143    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649146    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649149    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649151    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649154    2560 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649158    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649161    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:36.653981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649164    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649168    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649171    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649174    2560 flags.go:64] FLAG: --help="false"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649178    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649181    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649183    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649186    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649189    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649193    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649196    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649199    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649202    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649204    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649207    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649210    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649213    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649216    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649219    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649222    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649225    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649227    2560 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649230    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649233    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:36.654603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649236    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649241    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649244    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649247    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649250    2560 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649253    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649256    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649258    2560 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649263    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649268    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649271    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649275    2560 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649277    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649280    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649283    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649286    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649289    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649292    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649294    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649301    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649304    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649307    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649310    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:36.655205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649313    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649318    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649321    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649323    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649327    2560 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649329    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649332    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0358742f92d40cbf8"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649336    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649338    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649341    2560 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649344    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649347    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649351    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649354    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649357    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649360    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649363    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649367    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649370    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649374    2560 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649377    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649380    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649383    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649386    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649389    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649392    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:36.655885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649395    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649398    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649401    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649404    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649407    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649409    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649412    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649415    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649418    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649423    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649426    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649429    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649433    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649436    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649439    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649441    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649444    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649447    2560 flags.go:64] FLAG: --v="2"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649451    2560 flags.go:64] FLAG: --version="false"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649455    2560 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649459    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.649462    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649565    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649570    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.656531 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649573    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649577    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649580    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649583    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649585    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649588    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649590    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649593    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649596    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649598    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649601    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649603    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649606    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649609    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649613    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649616    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649619    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649622    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649625    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649628    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.657133 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649631    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649633    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649636    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649639    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649642    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649645    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649648    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649651    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649654    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649656    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649659    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649663    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649666    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649669    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649672    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649674    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649677    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649679    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649682    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649685    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.657655 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649688    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649690    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649693    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649695    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649698    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649700    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649703    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649706    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649708    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649712    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649732    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649735    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649738    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649754    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649756    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649759    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649762    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649765    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.658148 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649767    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649770    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649772    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649776    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649778    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649782    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649785    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649788    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649791    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649795    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649798    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649801    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649804    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649806    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649809    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649812    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649815    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649818    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649821    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649824    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.658587 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649827    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649830    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649832    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649835    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649838    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649841    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649850    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649854    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649857    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649860    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649863    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649866    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649868    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649871    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649874    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649877    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649881    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649885    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.659083 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.649888    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.650409    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.656831    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.656845    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.658349    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.658356    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658408    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658413    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658417    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658420    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658423    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658426    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658429    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658432    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658434    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.659520 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658437    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658440    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658442    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658446    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658448    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658451    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658454    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658456    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658459    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658462    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658464    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658467    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658470    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658473    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658475    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658479    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658482    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658486    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658490    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.659908 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658492    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658496    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658500    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658504    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658508    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658511    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658514    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658517    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658519    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658522    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658525    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658527    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658530    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658532    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658535    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658538    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658540    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658543    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658545    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.660379 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658548    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658550    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658553    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658555    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658558    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658561    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658563    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658566    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658569    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658571    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658574    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658577    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658579    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658582    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658585    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658587    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658590    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658593    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658596    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658599    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.660893 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658602    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658605    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658607    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658610    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658612    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658615    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658618    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658620    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658623    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658626    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658628    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658631    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658633    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658636    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658638    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658641    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658644    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658646    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658649    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658651    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.661373 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658654    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658656    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658659    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658662    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658664    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658667    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658670    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658672    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658675    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658677    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658680    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658682    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.658687    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658818    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658823    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.661878 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658826    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658829    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658832    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658834    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658837    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658840    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658842    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658846    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658849    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658852    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658854    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658857    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658859    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658861    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658865    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658869    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658872    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658875    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.662281 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658878    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658881    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658884    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658887    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658890    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658893    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658896    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658898    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658901    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658903    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658906    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658908    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658911    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658913    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658916    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658918    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658920    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658923    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658926    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658928    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.662761 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658931    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658933    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658936    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658938    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658940    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658943    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658945    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658948    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658951    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658953    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658955    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658958    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658960    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658963    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658966    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658969    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658971    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658974    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658977    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658979    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.663252 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658981    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658984    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658986    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658989    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658991    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658994    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658996    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.658999    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659001    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659006    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659009    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659011    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659014    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659016    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659019    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659021    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659024    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659026    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659028    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659031    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.663729 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659033    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659036    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659038    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659040    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659043    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659046    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659048    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659051    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659053    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659056    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659058    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659061    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659063    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659066    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659068    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659070    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659073    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659076    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.664230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:36.659078    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.659082    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.659169    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.661776    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.662606    2560 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.662712    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:36.664677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.662774    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:36.665370 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.665357    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:36.667301 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.667286    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:36.679771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.679738    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:36.682410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.682395    2560 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:36.683173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.683158    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:36.685964 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.685943    2560 fs.go:135] Filesystem UUIDs: map[0475651d-3c93-4fc6-8a7a-27d307d82e69:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7fe207e9-0d41-4ab9-95cd-211ffd576f04:/dev/nvme0n1p4]
Sep 05 02:17:36.686052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.685963    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:36.690973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.690866    2560 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:36.689544342 +0000 UTC m=+0.328756153 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095685 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec271921acf8510c1755fb679c79f22f SystemUUID:ec271921-acf8-510c-1755-fb679c79f22f BootID:30a4835d-4976-451e-8c6b-487a69e48549 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:fe:f8:df:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:fe:f8:df:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:08:74:77:04:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:36.690973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.690961    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:36.691128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.691050    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:36.693379 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.693349    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:36.693519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.693382    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:36.694618 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.694607    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:36.694651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.694621    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:36.694651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.694629    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:36.694944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.694929    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:36.695269 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.695258    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:36.696772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.696762    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:36.700387 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700367    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:36.700506 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700497    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:36.700534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700525    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:36.700561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700534    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:36.700561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700547    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:36.700561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700556    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:36.700653 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700564    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:36.700884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700871    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:36.700947 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.700891    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:36.703177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.703159    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:36.705147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705134    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:36.705197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705164    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:36.705327 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705316    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:36.705377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705331    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:36.705377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705338    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:36.705377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705343    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:36.705377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705350    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:36.705377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705356    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:36.705940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705928    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:36.705984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705941    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:36.705984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705952    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:36.705984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705960    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:36.705984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705973    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:36.705984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.705985    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:36.706884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.706872    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:36.706927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.706887    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:36.711680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.711656    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-13.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:36.711780 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.711710    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:36.711927 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.711908    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:36.712792 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.712778    2560 server.go:1268] "Started kubelet"
Sep 05 02:17:36.712877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.712834    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:36.712937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.712885    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:36.712992 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.712955    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:36.713693 ip-10-0-140-13 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:36.714442 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.714426    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:36.715549 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.715537    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:36.717398 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.717361    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:36.717794 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.717780    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:36.718423 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.717979    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:36.718423 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.718275    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:36.718923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.718781    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:36.719413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.719399    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:36.719413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.719414    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:36.719525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.719517    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:36.719525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.719525    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:36.720454 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720438    2560 factory.go:55] Registering systemd factory
Sep 05 02:17:36.720539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720473    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:36.720706 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720692    2560 factory.go:153] Registering CRI-O factory
Sep 05 02:17:36.720780 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720708    2560 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:36.720780 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720772    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:36.720884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720791    2560 factory.go:103] Registering Raw factory
Sep 05 02:17:36.720884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.720804    2560 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:36.721140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.721129    2560 manager.go:349] Starting recovery of all containers
Sep 05 02:17:36.723122 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.723094    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:17:36.723220 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.723176    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:17:36.723611 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.722545    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8ba0dd20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.71272784 +0000 UTC m=+0.351939650,LastTimestamp:2026-09-05 02:17:36.71272784 +0000 UTC m=+0.351939650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.730637 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.730620    2560 manager.go:354] Recovery completed
Sep 05 02:17:36.736514 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.736411    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.738420 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.738404    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.738489 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.738431    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.738489 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.738442    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.739017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739001    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.739080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739005    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:36.739080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739062    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:36.739139 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739081    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:36.739679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739662    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.739753 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739693    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.739753 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.739705    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.740663 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.740590    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d28e3ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738419642 +0000 UTC m=+0.377631433,LastTimestamp:2026-09-05 02:17:36.738419642 +0000 UTC m=+0.377631433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.749033 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.748963    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d2924b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738436276 +0000 UTC m=+0.377648067,LastTimestamp:2026-09-05 02:17:36.738436276 +0000 UTC m=+0.377648067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.756609 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.756550    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d294ac0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738446016 +0000 UTC m=+0.377657807,LastTimestamp:2026-09-05 02:17:36.738446016 +0000 UTC m=+0.377657807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.766650 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.766585    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-13.ec2.internal.18d24b2c8d28e3ba\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d28e3ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738419642 +0000 UTC m=+0.377631433,LastTimestamp:2026-09-05 02:17:36.739678932 +0000 UTC m=+0.378890723,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.769376 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.769361    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z7k5s"
Sep 05 02:17:36.777339 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.777286    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-13.ec2.internal.18d24b2c8d2924b4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d2924b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738436276 +0000 UTC m=+0.377648067,LastTimestamp:2026-09-05 02:17:36.739699485 +0000 UTC m=+0.378911276,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.777728 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.777713    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z7k5s"
Sep 05 02:17:36.787108 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.787055    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-13.ec2.internal.18d24b2c8d294ac0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-13.ec2.internal.18d24b2c8d294ac0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-13.ec2.internal,UID:ip-10-0-140-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-13.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-13.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.738446016 +0000 UTC m=+0.377657807,LastTimestamp:2026-09-05 02:17:36.739708721 +0000 UTC m=+0.378920511,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-13.ec2.internal,}"
Sep 05 02:17:36.819244 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.819228    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:36.889149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.889089    2560 policy_none.go:49] "None policy: Start"
Sep 05 02:17:36.889149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.889121    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:36.889149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.889137    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:36.890086 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.890073    2560 policy_none.go:47] "Start"
Sep 05 02:17:36.917239 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917226    2560 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:36.917364 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.917263    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:36.917364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917276    2560 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:36.917583 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917570    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:36.917645 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917585    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:36.917691 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917679    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:36.917787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917770    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:36.917787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.917785    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:36.918307 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.918285    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:36.918376 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.918333    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:36.933803 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.933786    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-13.ec2.internal\" not found" node="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:36.963139 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.963113    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:36.964322 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.964303    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:36.964402 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.964366    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:36.964402 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.964390    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:36.964482 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:36.964464    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:36.966761 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:36.966729    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:37.018704 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.018677    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.019483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.019467    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.019533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.019496    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.019533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.019506    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.019533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.019526    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.028667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.028655    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.028713 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.028672    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-13.ec2.internal\": node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.064782 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.064760    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"]
Sep 05 02:17:37.064845 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.064832    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.064889 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.064848    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.066107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.066093    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.066172 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.066119    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.066172 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.066129    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.067190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067178    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.067310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.067368 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067330    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.067860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067845    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.067915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067872    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.067915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067882    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.067988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067850    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.067988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067941    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.067988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.067953    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.069800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.069785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.069872 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.069810    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.070406 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.070391    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.070481 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.070416    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.070481 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.070426    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.088246 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.088227    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-13.ec2.internal\" not found" node="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.091473 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.091457    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-13.ec2.internal\" not found" node="ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.120665 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.120645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.120763 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.120669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.120763 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.120683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cd236a272d64976424037015e768d67-config\") pod \"kube-apiserver-proxy-ip-10-0-140-13.ec2.internal\" (UID: \"2cd236a272d64976424037015e768d67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.165904 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.165861    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.221252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.221340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cd236a272d64976424037015e768d67-config\") pod \"kube-apiserver-proxy-ip-10-0-140-13.ec2.internal\" (UID: \"2cd236a272d64976424037015e768d67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.221340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.221340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.221340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7695aad1513bda9b5fed55cb8af40fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal\" (UID: \"b7695aad1513bda9b5fed55cb8af40fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.221340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.221328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cd236a272d64976424037015e768d67-config\") pod \"kube-apiserver-proxy-ip-10-0-140-13.ec2.internal\" (UID: \"2cd236a272d64976424037015e768d67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.266345 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.266328    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.366948 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.366912    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.391071 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.391053    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.394759 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.394626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:37.468602 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.468543    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.570628 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.570592    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.665183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.665155    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:17:37.665532 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.665319    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:37.670747 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.670726    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.771032 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.770967    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.844642 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.844624    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:37.845802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.845780    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:37.845926 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.845819    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:37.845926 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.845842    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:37.871164 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.871135    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.890429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.890359    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:37 +0000 UTC" deadline="2028-04-24 00:00:46.354599809 +0000 UTC"
Sep 05 02:17:37.890429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.890397    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14325h43m8.464206456s"
Sep 05 02:17:37.971372 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:37.971345    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:37.972763 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:37.972736    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd236a272d64976424037015e768d67.slice/crio-68f39040409e6a7808fa922f8bc478e166816cb6add1db73f905901ca9cf1388 WatchSource:0}: Error finding container 68f39040409e6a7808fa922f8bc478e166816cb6add1db73f905901ca9cf1388: Status 404 returned error can't find the container with id 68f39040409e6a7808fa922f8bc478e166816cb6add1db73f905901ca9cf1388
Sep 05 02:17:37.972980 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:37.972960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7695aad1513bda9b5fed55cb8af40fb.slice/crio-9336b8d79e482cf097e605d75716b1163ffb35930c3729ad75091a5098a8b3a9 WatchSource:0}: Error finding container 9336b8d79e482cf097e605d75716b1163ffb35930c3729ad75091a5098a8b3a9: Status 404 returned error can't find the container with id 9336b8d79e482cf097e605d75716b1163ffb35930c3729ad75091a5098a8b3a9
Sep 05 02:17:37.976416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:37.976399    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:17:38.074897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.074835    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:38.153897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.153875    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:38.176048 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.176022    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-13.ec2.internal\" not found"
Sep 05 02:17:38.246322 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.246303    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:38.286384 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.286359    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:38.319948 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.319922    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:38.335680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.335629    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:38.336836 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.336822    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal"
Sep 05 02:17:38.349832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.349804    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:38.487651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.487617    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:38.702141 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.702060    2560 apiserver.go:52] "Watching apiserver"
Sep 05 02:17:38.709422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.709392    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:17:38.709984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.709958    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-r92cr","openshift-multus/multus-x2bng","openshift-multus/network-metrics-daemon-mj9x2","openshift-network-operator/iptables-alerter-cgmc8","kube-system/global-pull-secret-syncer-8zl8r","openshift-cluster-node-tuning-operator/tuned-mwksw","openshift-image-registry/node-ca-7n8lq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal","openshift-multus/multus-additional-cni-plugins-f2hw6","openshift-network-diagnostics/network-check-target-lb7h9","openshift-ovn-kubernetes/ovnkube-node-lncjl","kube-system/konnectivity-agent-bsckk","kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"]
Sep 05 02:17:38.711798 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.711773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.712947 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.712920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.714271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.714221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.714384 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.714300    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:38.716096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.715844    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.716278    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.716553    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.716734    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.716859    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.716944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.717074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vjtz8\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.717217    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.717391    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76mrd\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.717446    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.717625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.718459    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.718677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.719275    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:38.720235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.720089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.720977 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.720340    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:17:38.721211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.721187    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sblq5\""
Sep 05 02:17:38.721500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.721484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.721579 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.721549    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:38.723946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.723924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.724910 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.724893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.726320 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.726283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.728152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.727825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.729138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:38.729233 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.729181    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:38.729682 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.729756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-ovn\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.729756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-script-lib\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.729891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.729891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-iptables-alerter-script\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.729891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-env-overrides\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.729891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmsq\" (UniqueName: \"kubernetes.io/projected/fcc83524-5eb0-4089-8aae-08ff8edd38c6-kube-api-access-qxmsq\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.729891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-host\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.730147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.729908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9gl\" (UniqueName: \"kubernetes.io/projected/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-kube-api-access-4t9gl\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.730200 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.730170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.733736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.730098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-systemd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.733855 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-etc-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.733855 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-log-socket\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.733980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-var-lib-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.733980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-dbus\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.733980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-serviceca\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.733980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-slash\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.733999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kvr\" (UniqueName: \"kubernetes.io/projected/72c887cb-e8a9-4efc-83cd-46f1220a2c52-kube-api-access-57kvr\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-kube-api-access-5sm8n\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-systemd-units\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-bin\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-netd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-config\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovn-node-metrics-cert\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-netns\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-node-log\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-kubelet-config\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-kubelet\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734437    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:17:38.734503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.735017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.734745    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.735017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.735008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-host-slash\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.735243 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.735224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.735392 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.735371    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ctzzl\""
Sep 05 02:17:38.736044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736021    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.736135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736087    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:17:38.736410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmvnt\""
Sep 05 02:17:38.736704 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.736907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736889    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.737038 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736909    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:17:38.737038 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.736932    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c7jxq\""
Sep 05 02:17:38.737298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.737280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:17:38.737362 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.737313    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.737531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.737508    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c8jfl\""
Sep 05 02:17:38.737587 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.737281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:17:38.742375 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.742353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.742475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.742389    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.742475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.742400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tjnt6\""
Sep 05 02:17:38.742475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.742447    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.746208 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.746185    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:38.746491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.746472    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:17:38.746992 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.746971    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:17:38.747223 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.747207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4g8p2\""
Sep 05 02:17:38.769110 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.769085    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvvlp"
Sep 05 02:17:38.780075 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.780032    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvvlp"
Sep 05 02:17:38.819987 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.819936    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:17:38.835479 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.835579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-ovn\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.835579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac596e9e-cfc7-477a-bf81-6541df54e1fd-agent-certs\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.835579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb62105b-3282-459d-a241-fb10b3ae0191-hosts-file\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.835579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-tuned\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-ovn\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmsq\" (UniqueName: \"kubernetes.io/projected/fcc83524-5eb0-4089-8aae-08ff8edd38c6-kube-api-access-qxmsq\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-host\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-host\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9gl\" (UniqueName: \"kubernetes.io/projected/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-kube-api-access-4t9gl\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-systemd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.835821 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-etc-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-sys-fs\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-var-lib-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-run-systemd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-dbus\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-netns\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.835993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-dbus\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-conf-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-etc-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-var-lib-openvswitch\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb62105b-3282-459d-a241-fb10b3ae0191-tmp-dir\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblts\" (UniqueName: \"kubernetes.io/projected/fb62105b-3282-459d-a241-fb10b3ae0191-kube-api-access-nblts\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-systemd\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-tmp\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.836231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-serviceca\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sdp\" (UniqueName: \"kubernetes.io/projected/315f69fb-bac2-49cc-b31a-a66da307c403-kube-api-access-47sdp\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-lib-modules\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-socket-dir-parent\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-system-cni-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-systemd-units\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-bin\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-netd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-systemd-units\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-config\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-system-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-multus-daemon-config\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836612    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-registration-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-node-log\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-cnibin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-modprobe-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-host-slash\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-kubernetes\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425q4\" (UniqueName: \"kubernetes.io/projected/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-kube-api-access-425q4\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-script-lib\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.836985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-iptables-alerter-script\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-bin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsgm\" (UniqueName: \"kubernetes.io/projected/5e624dab-2c86-455c-a2f0-78752f63e187-kube-api-access-lxsgm\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-conf\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-env-overrides\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-log-socket\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-config\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.837883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-device-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-multus-certs\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-run\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-host\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-slash\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-netd\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kvr\" (UniqueName: \"kubernetes.io/projected/72c887cb-e8a9-4efc-83cd-46f1220a2c52-kube-api-access-57kvr\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-serviceca\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-cni-binary-copy\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-node-log\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.837474    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-host-slash\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-cni-bin\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.837532    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.337511579 +0000 UTC m=+2.976723365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-slash\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-cnibin\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-log-socket\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.838736 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.837804    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.837850    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.337839687 +0000 UTC m=+2.977051467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djh4\" (UniqueName: \"kubernetes.io/projected/ff423175-7838-47e6-976f-61484063f3d8-kube-api-access-5djh4\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-kube-api-access-5sm8n\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-k8s-cni-cncf-io\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-hostroot\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.837969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-env-overrides\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysconfig\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-iptables-alerter-script\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-var-lib-kubelet\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-multus\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-kubelet\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac596e9e-cfc7-477a-bf81-6541df54e1fd-konnectivity-ca\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.839702 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovn-node-metrics-cert\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-netns\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-kubelet-config\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-sys\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-kubelet\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-etc-kubernetes\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovnkube-script-lib\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-socket-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda2fbf9-4539-4dec-afee-46b0f5635648-kubelet-config\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-kubelet\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c887cb-e8a9-4efc-83cd-46f1220a2c52-host-run-netns\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.840505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.838750    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:17:38.844214 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.844189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72c887cb-e8a9-4efc-83cd-46f1220a2c52-ovn-node-metrics-cert\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.848668 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.848645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmsq\" (UniqueName: \"kubernetes.io/projected/fcc83524-5eb0-4089-8aae-08ff8edd38c6-kube-api-access-qxmsq\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:38.849909 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.849883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kvr\" (UniqueName: \"kubernetes.io/projected/72c887cb-e8a9-4efc-83cd-46f1220a2c52-kube-api-access-57kvr\") pod \"ovnkube-node-lncjl\" (UID: \"72c887cb-e8a9-4efc-83cd-46f1220a2c52\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:38.850176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.850156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/cf2e9965-c0c1-4682-9359-b5a22dbc0f89-kube-api-access-5sm8n\") pod \"iptables-alerter-cgmc8\" (UID: \"cf2e9965-c0c1-4682-9359-b5a22dbc0f89\") " pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:38.850240 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.850201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9gl\" (UniqueName: \"kubernetes.io/projected/b29d312f-67d2-4530-bcfd-b6e7f1dada9c-kube-api-access-4t9gl\") pod \"node-ca-7n8lq\" (UID: \"b29d312f-67d2-4530-bcfd-b6e7f1dada9c\") " pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:38.938902 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.938867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-systemd\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.938902 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.938904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-tmp\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.938926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sdp\" (UniqueName: \"kubernetes.io/projected/315f69fb-bac2-49cc-b31a-a66da307c403-kube-api-access-47sdp\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.938954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-lib-modules\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.938978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-socket-dir-parent\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-system-cni-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-system-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-multus-daemon-config\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-registration-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-cnibin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:38.939154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-lib-modules\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-system-cni-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-modprobe-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-socket-dir-parent\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-kubernetes\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-modprobe-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-registration-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425q4\" (UniqueName: \"kubernetes.io/projected/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-kube-api-access-425q4\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-cnibin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-bin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsgm\" (UniqueName: \"kubernetes.io/projected/5e624dab-2c86-455c-a2f0-78752f63e187-kube-api-access-lxsgm\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-conf\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-device-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-kubernetes\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-bin\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.939800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-device-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-system-cni-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-conf\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-systemd\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-multus-certs\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-multus-certs\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-run\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-run\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-host\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-host\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-cni-binary-copy\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939877    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-cnibin\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djh4\" (UniqueName: \"kubernetes.io/projected/ff423175-7838-47e6-976f-61484063f3d8-kube-api-access-5djh4\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-multus-daemon-config\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-k8s-cni-cncf-io\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e624dab-2c86-455c-a2f0-78752f63e187-cnibin\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-hostroot\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.940687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.939993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-k8s-cni-cncf-io\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysconfig\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-var-lib-kubelet\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940086    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-multus\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-hostroot\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-kubelet\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac596e9e-cfc7-477a-bf81-6541df54e1fd-konnectivity-ca\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-cni-multus\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-var-lib-kubelet\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysconfig\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-sys\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-etc-kubernetes\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-sys\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940364    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/315f69fb-bac2-49cc-b31a-a66da307c403-cni-binary-copy\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.941744 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-etc-kubernetes\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-socket-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac596e9e-cfc7-477a-bf81-6541df54e1fd-agent-certs\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb62105b-3282-459d-a241-fb10b3ae0191-hosts-file\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-tuned\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-sys-fs\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-socket-dir\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-netns\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-host-run-netns\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb62105b-3282-459d-a241-fb10b3ae0191-hosts-file\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-conf-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb62105b-3282-459d-a241-fb10b3ae0191-tmp-dir\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblts\" (UniqueName: \"kubernetes.io/projected/fb62105b-3282-459d-a241-fb10b3ae0191-kube-api-access-nblts\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.943741 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-var-lib-kubelet\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e624dab-2c86-455c-a2f0-78752f63e187-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-sysctl-d\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/315f69fb-bac2-49cc-b31a-a66da307c403-multus-conf-dir\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.940991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff423175-7838-47e6-976f-61484063f3d8-sys-fs\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.941028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb62105b-3282-459d-a241-fb10b3ae0191-tmp-dir\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.941053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac596e9e-cfc7-477a-bf81-6541df54e1fd-konnectivity-ca\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.942459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-tmp\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.944555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.943263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-etc-tuned\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.945595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.945572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac596e9e-cfc7-477a-bf81-6541df54e1fd-agent-certs\") pod \"konnectivity-agent-bsckk\" (UID: \"ac596e9e-cfc7-477a-bf81-6541df54e1fd\") " pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:38.947013 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.946717    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:38.947013 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.946744    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:38.947013 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.946772    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:38.947013 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:38.946833    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.446814113 +0000 UTC m=+3.086025906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:38.949343 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.949318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-425q4\" (UniqueName: \"kubernetes.io/projected/104bcdef-0a06-49f0-b2f8-6fadfc3950a4-kube-api-access-425q4\") pod \"tuned-mwksw\" (UID: \"104bcdef-0a06-49f0-b2f8-6fadfc3950a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:38.949721 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.949697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sdp\" (UniqueName: \"kubernetes.io/projected/315f69fb-bac2-49cc-b31a-a66da307c403-kube-api-access-47sdp\") pod \"multus-x2bng\" (UID: \"315f69fb-bac2-49cc-b31a-a66da307c403\") " pod="openshift-multus/multus-x2bng"
Sep 05 02:17:38.950197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.950171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsgm\" (UniqueName: \"kubernetes.io/projected/5e624dab-2c86-455c-a2f0-78752f63e187-kube-api-access-lxsgm\") pod \"multus-additional-cni-plugins-f2hw6\" (UID: \"5e624dab-2c86-455c-a2f0-78752f63e187\") " pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:38.950695 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.950672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblts\" (UniqueName: \"kubernetes.io/projected/fb62105b-3282-459d-a241-fb10b3ae0191-kube-api-access-nblts\") pod \"node-resolver-r92cr\" (UID: \"fb62105b-3282-459d-a241-fb10b3ae0191\") " pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:38.952212 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.952153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djh4\" (UniqueName: \"kubernetes.io/projected/ff423175-7838-47e6-976f-61484063f3d8-kube-api-access-5djh4\") pod \"aws-ebs-csi-driver-node-swnk7\" (UID: \"ff423175-7838-47e6-976f-61484063f3d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:38.969955 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.969907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal" event={"ID":"2cd236a272d64976424037015e768d67","Type":"ContainerStarted","Data":"68f39040409e6a7808fa922f8bc478e166816cb6add1db73f905901ca9cf1388"}
Sep 05 02:17:38.971477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:38.971444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal" event={"ID":"b7695aad1513bda9b5fed55cb8af40fb","Type":"ContainerStarted","Data":"9336b8d79e482cf097e605d75716b1163ffb35930c3729ad75091a5098a8b3a9"}
Sep 05 02:17:39.027435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.027409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7n8lq"
Sep 05 02:17:39.042668 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.042644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:17:39.054460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.054437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cgmc8"
Sep 05 02:17:39.061421 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.061402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwksw"
Sep 05 02:17:39.068903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.068880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2hw6"
Sep 05 02:17:39.077624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.077604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7"
Sep 05 02:17:39.087122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.087097    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2bng"
Sep 05 02:17:39.095773 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.095754    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r92cr"
Sep 05 02:17:39.103764 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.103744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:17:39.344341 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.344257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:39.344341 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.344314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:39.344552 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.344429    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:39.344552 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.344492    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.34447253 +0000 UTC m=+3.983684314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:39.344552 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.344431    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:39.344695 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.344599    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.344576272 +0000 UTC m=+3.983788064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:39.545482 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.545448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:39.545642 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.545624    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:39.545708 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.545646    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:39.545708 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.545657    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:39.545819 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.545708    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.54569381 +0000 UTC m=+4.184905589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:39.783106 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.783002    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:39 +0000 UTC" deadline="2028-06-03 20:07:04.734777922 +0000 UTC"
Sep 05 02:17:39.783106 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.783055    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15305h49m24.951729179s"
Sep 05 02:17:39.965701 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.965670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:39.965897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.965670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:39.965897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:39.965813    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:39.965897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.965809    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:39.966060 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.965916    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:39.966060 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:39.966013    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:40.054478 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.054444    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29d312f_67d2_4530_bcfd_b6e7f1dada9c.slice/crio-d8cdf8b205c6045aa189f6d26cfd99d0257e9974b90282ad065d75d1103be684 WatchSource:0}: Error finding container d8cdf8b205c6045aa189f6d26cfd99d0257e9974b90282ad065d75d1103be684: Status 404 returned error can't find the container with id d8cdf8b205c6045aa189f6d26cfd99d0257e9974b90282ad065d75d1103be684
Sep 05 02:17:40.056100 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.056078    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb62105b_3282_459d_a241_fb10b3ae0191.slice/crio-f66249388d3eb9d95167d10331088097796e70b08d68d63daafcd2aed6d1c494 WatchSource:0}: Error finding container f66249388d3eb9d95167d10331088097796e70b08d68d63daafcd2aed6d1c494: Status 404 returned error can't find the container with id f66249388d3eb9d95167d10331088097796e70b08d68d63daafcd2aed6d1c494
Sep 05 02:17:40.072425 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.072239    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315f69fb_bac2_49cc_b31a_a66da307c403.slice/crio-95bdfcc68149d838812f8ca88fbd889734eb518dfeb71ff77ca5aa59d79596e1 WatchSource:0}: Error finding container 95bdfcc68149d838812f8ca88fbd889734eb518dfeb71ff77ca5aa59d79596e1: Status 404 returned error can't find the container with id 95bdfcc68149d838812f8ca88fbd889734eb518dfeb71ff77ca5aa59d79596e1
Sep 05 02:17:40.074033 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.074012    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c887cb_e8a9_4efc_83cd_46f1220a2c52.slice/crio-27523c9a065e989e05e4e50fccc9ef46d8574b3ba298c8b80f30a1efc69ad33f WatchSource:0}: Error finding container 27523c9a065e989e05e4e50fccc9ef46d8574b3ba298c8b80f30a1efc69ad33f: Status 404 returned error can't find the container with id 27523c9a065e989e05e4e50fccc9ef46d8574b3ba298c8b80f30a1efc69ad33f
Sep 05 02:17:40.074804 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.074785    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104bcdef_0a06_49f0_b2f8_6fadfc3950a4.slice/crio-5e883b6bcd88a3779677a6180ea484d8ed7720a783a04cd90b275cc25c835790 WatchSource:0}: Error finding container 5e883b6bcd88a3779677a6180ea484d8ed7720a783a04cd90b275cc25c835790: Status 404 returned error can't find the container with id 5e883b6bcd88a3779677a6180ea484d8ed7720a783a04cd90b275cc25c835790
Sep 05 02:17:40.076525 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.076427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff423175_7838_47e6_976f_61484063f3d8.slice/crio-84516215782ec647d9c764f3c009243cc2b3295b8aeec4b08f3b2ecc313ac064 WatchSource:0}: Error finding container 84516215782ec647d9c764f3c009243cc2b3295b8aeec4b08f3b2ecc313ac064: Status 404 returned error can't find the container with id 84516215782ec647d9c764f3c009243cc2b3295b8aeec4b08f3b2ecc313ac064
Sep 05 02:17:40.077169 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.077147    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e624dab_2c86_455c_a2f0_78752f63e187.slice/crio-c7e047dec23da168cc91f5b7e3405ae499df3123537cee429a5259f0f13d5d7d WatchSource:0}: Error finding container c7e047dec23da168cc91f5b7e3405ae499df3123537cee429a5259f0f13d5d7d: Status 404 returned error can't find the container with id c7e047dec23da168cc91f5b7e3405ae499df3123537cee429a5259f0f13d5d7d
Sep 05 02:17:40.079577 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.079556    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2e9965_c0c1_4682_9359_b5a22dbc0f89.slice/crio-9a90474e5fdcaa303ea4a3fe362c0d1378948f87330d18967afe830b509a1055 WatchSource:0}: Error finding container 9a90474e5fdcaa303ea4a3fe362c0d1378948f87330d18967afe830b509a1055: Status 404 returned error can't find the container with id 9a90474e5fdcaa303ea4a3fe362c0d1378948f87330d18967afe830b509a1055
Sep 05 02:17:40.081172 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:17:40.081096    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac596e9e_cfc7_477a_bf81_6541df54e1fd.slice/crio-2eea4b2c013192fe5bb29c27641d37f8de419a22766a4896aa58c9cbb33197c8 WatchSource:0}: Error finding container 2eea4b2c013192fe5bb29c27641d37f8de419a22766a4896aa58c9cbb33197c8: Status 404 returned error can't find the container with id 2eea4b2c013192fe5bb29c27641d37f8de419a22766a4896aa58c9cbb33197c8
Sep 05 02:17:40.351442 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.351412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:40.351539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.351457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:40.351584 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.351546    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:40.351584 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.351549    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:40.351649 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.351604    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.351585177 +0000 UTC m=+5.990796963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:40.351649 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.351621    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.351612148 +0000 UTC m=+5.990823935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:40.553907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.553870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:40.554103 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.554070    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:40.554103 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.554102    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:40.554300 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.554116    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:40.554300 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:40.554187    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.554158681 +0000 UTC m=+6.193370465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:40.783551 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.783428    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:39 +0000 UTC" deadline="2028-06-14 14:09:16.157656136 +0000 UTC"
Sep 05 02:17:40.783551 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.783502    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15563h51m35.374158596s"
Sep 05 02:17:40.986463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:40.985939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bsckk" event={"ID":"ac596e9e-cfc7-477a-bf81-6541df54e1fd","Type":"ContainerStarted","Data":"2eea4b2c013192fe5bb29c27641d37f8de419a22766a4896aa58c9cbb33197c8"}
Sep 05 02:17:41.011497 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.011463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerStarted","Data":"c7e047dec23da168cc91f5b7e3405ae499df3123537cee429a5259f0f13d5d7d"}
Sep 05 02:17:41.017977 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.017949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7" event={"ID":"ff423175-7838-47e6-976f-61484063f3d8","Type":"ContainerStarted","Data":"84516215782ec647d9c764f3c009243cc2b3295b8aeec4b08f3b2ecc313ac064"}
Sep 05 02:17:41.025828 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.025789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwksw" event={"ID":"104bcdef-0a06-49f0-b2f8-6fadfc3950a4","Type":"ContainerStarted","Data":"5e883b6bcd88a3779677a6180ea484d8ed7720a783a04cd90b275cc25c835790"}
Sep 05 02:17:41.036239 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.036170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"27523c9a065e989e05e4e50fccc9ef46d8574b3ba298c8b80f30a1efc69ad33f"}
Sep 05 02:17:41.047218 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.047188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r92cr" event={"ID":"fb62105b-3282-459d-a241-fb10b3ae0191","Type":"ContainerStarted","Data":"f66249388d3eb9d95167d10331088097796e70b08d68d63daafcd2aed6d1c494"}
Sep 05 02:17:41.068936 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.067971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal" event={"ID":"2cd236a272d64976424037015e768d67","Type":"ContainerStarted","Data":"97d8142c5a342ba4c82e2a79a58965db44d8dd883044739223b836bcdae16e1c"}
Sep 05 02:17:41.090314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.089810    2560 generic.go:356] "Generic (PLEG): container finished" podID="b7695aad1513bda9b5fed55cb8af40fb" containerID="20b2e870af8cbc5506c086dcc37ad18299db13d030c52ccc642099b61829f2c4" exitCode=0
Sep 05 02:17:41.090314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.089923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal" event={"ID":"b7695aad1513bda9b5fed55cb8af40fb","Type":"ContainerDied","Data":"20b2e870af8cbc5506c086dcc37ad18299db13d030c52ccc642099b61829f2c4"}
Sep 05 02:17:41.098551 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.098368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cgmc8" event={"ID":"cf2e9965-c0c1-4682-9359-b5a22dbc0f89","Type":"ContainerStarted","Data":"9a90474e5fdcaa303ea4a3fe362c0d1378948f87330d18967afe830b509a1055"}
Sep 05 02:17:41.102195 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.102122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2bng" event={"ID":"315f69fb-bac2-49cc-b31a-a66da307c403","Type":"ContainerStarted","Data":"95bdfcc68149d838812f8ca88fbd889734eb518dfeb71ff77ca5aa59d79596e1"}
Sep 05 02:17:41.105201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.105141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7n8lq" event={"ID":"b29d312f-67d2-4530-bcfd-b6e7f1dada9c","Type":"ContainerStarted","Data":"d8cdf8b205c6045aa189f6d26cfd99d0257e9974b90282ad065d75d1103be684"}
Sep 05 02:17:41.109268 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.108343    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-13.ec2.internal" podStartSLOduration=3.108324972 podStartE2EDuration="3.108324972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:41.086014239 +0000 UTC m=+4.725226044" watchObservedRunningTime="2026-09-05 02:17:41.108324972 +0000 UTC m=+4.747536779"
Sep 05 02:17:41.964639 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.964603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:41.964639 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.964638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:41.965342 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:41.964751    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:41.965342 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:41.964843    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:41.965342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:41.964918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:41.965342 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:41.965021    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:42.133984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:42.133898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal" event={"ID":"b7695aad1513bda9b5fed55cb8af40fb","Type":"ContainerStarted","Data":"6ad1747db199946192905f5a02d46314ff9ec78d336bb9225ec73b7a98e7d802"}
Sep 05 02:17:42.368758 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:42.368722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:42.368940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:42.368798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:42.368940 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.368935    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.369072 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.368995    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.368977891 +0000 UTC m=+10.008189676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.369072 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.369068    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.369188 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.369104    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.36909262 +0000 UTC m=+10.008304407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.570195 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:42.570110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:42.570355 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.570302    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:42.570355 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.570327    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:42.570355 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.570342    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.570523 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:42.570398    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.570378985 +0000 UTC m=+10.209590769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:43.965436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:43.965396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:43.965436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:43.965432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:43.966003 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:43.965550    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:43.966003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:43.965915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:43.966121 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:43.966015    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:43.966121 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:43.966109    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:45.965421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:45.965548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:45.965562    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:45.965589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:45.965710    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:45.965963 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:45.965821    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:46.407813 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:46.407613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:46.407813 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:46.407675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:46.407813 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.407798    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:46.408084 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.407867    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.407848023 +0000 UTC m=+18.047059814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:46.408201 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.407803    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:46.408263 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.408228    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.408213887 +0000 UTC m=+18.047425667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:46.609258 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:46.609217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:46.609423 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.609409    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:46.609501 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.609433    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:46.609501 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.609447    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:46.609663 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:46.609510    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.609492239 +0000 UTC m=+18.248704041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:47.965060 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:47.964853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:47.965517 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:47.965079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:47.965517 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:47.965157    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:47.965640 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:47.965564    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:47.965640 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:47.965634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:47.965759 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:47.965714    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:49.965475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:49.965435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:49.965944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:49.965544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:49.965944 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:49.965548    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:49.965944 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:49.965647    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:49.965944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:49.965548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:49.965944 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:49.965737    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:51.965076 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:51.965041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:51.965526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:51.965052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:51.965526 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:51.965148    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:51.965526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:51.965179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:51.965526 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:51.965303    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:51.965526 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:51.965405    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:53.964589 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:53.964550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:53.965104 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:53.964598    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:53.965104 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:53.964686    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:53.965104 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:53.964776    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:53.965104 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:53.964791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:53.965104 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:53.964863    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:54.467980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:54.467943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:54.468163 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:54.468008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:54.468163 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.468093    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.468163 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.468104    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:54.468303 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.468166    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.468143687 +0000 UTC m=+34.107355472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.468303 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.468187    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.468176755 +0000 UTC m=+34.107388546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:54.670291 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:54.670260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:54.670477 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.670453    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:54.670534 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.670483    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:54.670534 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.670497    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:54.670619 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:54.670557    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.670538447 +0000 UTC m=+34.309750228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:55.965523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:55.965481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:55.965523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:55.965510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:55.966055 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:55.965496    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:55.966055 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:55.965600    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:55.966055 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:55.965679    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:55.966055 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:55.965761    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:57.965251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:57.965218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:57.965251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:57.965258    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:57.965757 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:57.965340    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:57.965757 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:57.965389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:57.965757 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:57.965465    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:57.965757 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:57.965560    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:17:59.171307 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.171054    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bsckk" event={"ID":"ac596e9e-cfc7-477a-bf81-6541df54e1fd","Type":"ContainerStarted","Data":"1d0394994a4e6dd57a5474dcc371e67f82c94e7bcacd0c5f5730db234f47d3e3"}
Sep 05 02:17:59.172709 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.172687    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="c9d320f82bf4cff86fb7a1af17b82d42860c4a9d70109b7c9311c9d90df83a7e" exitCode=0
Sep 05 02:17:59.172823 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.172788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"c9d320f82bf4cff86fb7a1af17b82d42860c4a9d70109b7c9311c9d90df83a7e"}
Sep 05 02:17:59.174458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.174437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7" event={"ID":"ff423175-7838-47e6-976f-61484063f3d8","Type":"ContainerStarted","Data":"bad8fa3703f7a185e82ad6b4bd7ef16031116b7cf8d7e49a6e48363480dcbd4a"}
Sep 05 02:17:59.175915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.175893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwksw" event={"ID":"104bcdef-0a06-49f0-b2f8-6fadfc3950a4","Type":"ContainerStarted","Data":"50d4f1ac3af93eb407b133d63772a18234f961d50d0cb921b789e1a0f2b98043"}
Sep 05 02:17:59.178635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.178619    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:17:59.179002 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.178979    2560 generic.go:356] "Generic (PLEG): container finished" podID="72c887cb-e8a9-4efc-83cd-46f1220a2c52" containerID="de48d6006802b46ec7d3156f5d58e1c83aef905a438f8eb3c771f873aee0004e" exitCode=1
Sep 05 02:17:59.179098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"e2d098efddaa15ad26d08b8c00d58bef51de574da852d2b7fc58f71e687724e5"}
Sep 05 02:17:59.179163 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"f0f96f79f29452b6b720c44c3cecf8745d8f9bf0415fca23dec76d9cb4a0adc8"}
Sep 05 02:17:59.179214 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"2084cd9450b1a6f354ab37e74a370afed12be97c675103c36a4e56fe6671d5b8"}
Sep 05 02:17:59.179262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"96a70c9e69009cb627e90adbad8d3131fdd05462acb4f40824c8fb846fc35763"}
Sep 05 02:17:59.179315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerDied","Data":"de48d6006802b46ec7d3156f5d58e1c83aef905a438f8eb3c771f873aee0004e"}
Sep 05 02:17:59.179367 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.179327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"bd230c91b670e2b1f3ca144964f2b4e20aa084a939f4aa4b2d062c839252d39b"}
Sep 05 02:17:59.180674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.180646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r92cr" event={"ID":"fb62105b-3282-459d-a241-fb10b3ae0191","Type":"ContainerStarted","Data":"178a04a84c019682a0b258829cf0369167da6846329105e38d7de60683cb4832"}
Sep 05 02:17:59.182180 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.182161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2bng" event={"ID":"315f69fb-bac2-49cc-b31a-a66da307c403","Type":"ContainerStarted","Data":"79670555e7dd75fc50d20cf092508a1377aec1aa868a99deec2f0dabd512eec5"}
Sep 05 02:17:59.183587 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.183566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7n8lq" event={"ID":"b29d312f-67d2-4530-bcfd-b6e7f1dada9c","Type":"ContainerStarted","Data":"0c6095d2466137af253fdf8517fc7a0e3b010b07e19af53f046cb2e3a7340403"}
Sep 05 02:17:59.188018 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.187981    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bsckk" podStartSLOduration=3.876127191 podStartE2EDuration="22.187969802s" totalImagesPullingTime="18.311842611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:59.187953043 +0000 UTC m=+22.827164854" watchObservedRunningTime="2026-09-05 02:17:59.187969802 +0000 UTC m=+22.827181602"
Sep 05 02:17:59.188434 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.188405    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-13.ec2.internal" podStartSLOduration=21.188398694 podStartE2EDuration="21.188398694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:42.152841003 +0000 UTC m=+5.792052835" watchObservedRunningTime="2026-09-05 02:17:59.188398694 +0000 UTC m=+22.827610492"
Sep 05 02:17:59.202768 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.202718    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7n8lq" podStartSLOduration=3.836850938 podStartE2EDuration="22.202706976s" totalImagesPullingTime="18.365856038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:59.202461508 +0000 UTC m=+22.841673327" watchObservedRunningTime="2026-09-05 02:17:59.202706976 +0000 UTC m=+22.841918775"
Sep 05 02:17:59.257731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.257695    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mwksw" podStartSLOduration=3.902708495 podStartE2EDuration="22.257682871s" totalImagesPullingTime="18.354974376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:59.257677411 +0000 UTC m=+22.896889229" watchObservedRunningTime="2026-09-05 02:17:59.257682871 +0000 UTC m=+22.896894671"
Sep 05 02:17:59.258146 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.258124    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x2bng" podStartSLOduration=3.488011595 podStartE2EDuration="22.258115918s" totalImagesPullingTime="18.770104323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:59.241338142 +0000 UTC m=+22.880549999" watchObservedRunningTime="2026-09-05 02:17:59.258115918 +0000 UTC m=+22.897327717"
Sep 05 02:17:59.965500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.965259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:17:59.965500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.965264    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:17:59.965693 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:59.965515    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:17:59.965693 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:17:59.965264    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:17:59.965693 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:59.965568    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:17:59.965693 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:17:59.965667    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:00.133114 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.133077    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:00.188069 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.188039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cgmc8" event={"ID":"cf2e9965-c0c1-4682-9359-b5a22dbc0f89","Type":"ContainerStarted","Data":"83c270bcedd8e57936cd4b16a9730522f94f1207500d925a0dfb227c9d4e45b0"}
Sep 05 02:18:00.190168 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.190141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7" event={"ID":"ff423175-7838-47e6-976f-61484063f3d8","Type":"ContainerStarted","Data":"04bb98eb960d98bbd93206697d4cd167fdad1868643057d9796d6d6a0b847822"}
Sep 05 02:18:00.203520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.203481    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r92cr" podStartSLOduration=4.875516259 podStartE2EDuration="23.203468204s" totalImagesPullingTime="18.327951945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:59.272692926 +0000 UTC m=+22.911904744" watchObservedRunningTime="2026-09-05 02:18:00.203468204 +0000 UTC m=+23.842680003"
Sep 05 02:18:00.203671 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.203651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cgmc8" podStartSLOduration=4.847971286 podStartE2EDuration="23.203644008s" totalImagesPullingTime="18.355672722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:00.203088675 +0000 UTC m=+23.842300477" watchObservedRunningTime="2026-09-05 02:18:00.203644008 +0000 UTC m=+23.842855808"
Sep 05 02:18:00.932433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.932309    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:00.133106424Z","UUID":"a0e95490-fed5-4496-ba17-fbdfe12a2912","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:00.934291 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.934264    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:00.934291 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:00.934299    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:01.964907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:01.964880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:01.965360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:01.964880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:01.965360 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:01.965012    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:01.965360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:01.964880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:01.965360 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:01.965048    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:01.965360 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:01.965144    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:02.197515 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:02.197482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7" event={"ID":"ff423175-7838-47e6-976f-61484063f3d8","Type":"ContainerStarted","Data":"e23e9d02ee71e848c2192df35bed694bdc26cede574d1ea56bcb422dd7de0385"}
Sep 05 02:18:02.201194 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:02.201170    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:18:02.201606 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:02.201584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"fb0243270286ffcf0ffed6d3d7675dbdda3b84a7caab68d8811ec24dbaf2e04a"}
Sep 05 02:18:02.216551 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:02.216479    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swnk7" podStartSLOduration=4.31677538 podStartE2EDuration="25.216465077s" totalImagesPullingTime="20.899689697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:02.215916209 +0000 UTC m=+25.855128031" watchObservedRunningTime="2026-09-05 02:18:02.216465077 +0000 UTC m=+25.855676933"
Sep 05 02:18:03.965143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:03.964977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:03.965143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:03.964988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:03.965768 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:03.964988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:03.965768 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:03.965238    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:03.965768 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:03.965292    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:03.965768 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:03.965368    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:03.972195 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:03.972175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:18:03.973075 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:03.973060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:18:04.208371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.208343    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="e1196900ffdbc4a92d2e94d9decd81d8dfbba275bb26372692e5097a265bfc80" exitCode=0
Sep 05 02:18:04.208479 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.208435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"e1196900ffdbc4a92d2e94d9decd81d8dfbba275bb26372692e5097a265bfc80"}
Sep 05 02:18:04.212204 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.212188    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:18:04.212624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.212602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"445872a269ca27129be230a218e03161651dea288a318aaa2fd68bfa6f4f7816"}
Sep 05 02:18:04.212789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.212768    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:18:04.213024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.213000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:04.213234 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.213083    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:04.213234 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.213156    2560 scope.go:117] "RemoveContainer" containerID="de48d6006802b46ec7d3156f5d58e1c83aef905a438f8eb3c771f873aee0004e"
Sep 05 02:18:04.213234 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.213225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bsckk"
Sep 05 02:18:04.228622 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:04.228605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:05.218831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.218807    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:18:05.219262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.219238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" event={"ID":"72c887cb-e8a9-4efc-83cd-46f1220a2c52","Type":"ContainerStarted","Data":"437b27df05f3e5a4cbfb3eef6492ed415fb696553c59f907e1c0dea4ab922b2d"}
Sep 05 02:18:05.219670 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.219648    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:05.236570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.236548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:05.248568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.248513    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl" podStartSLOduration=9.868501416 podStartE2EDuration="28.248494515s" totalImagesPullingTime="18.379993099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:05.247882211 +0000 UTC m=+28.887094034" watchObservedRunningTime="2026-09-05 02:18:05.248494515 +0000 UTC m=+28.887706316"
Sep 05 02:18:05.508057 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.507855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zl8r"]
Sep 05 02:18:05.508150 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.508130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:05.508662 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:05.508635    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:05.511340 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.511320    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lb7h9"]
Sep 05 02:18:05.511464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.511408    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:05.511530 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:05.511487    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:05.511910 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.511887    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mj9x2"]
Sep 05 02:18:05.512009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:05.511992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:05.512107 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:05.512080    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:06.223677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:06.223638    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="34a9133e817a564e573345fb80126e7d804dc724b9295958c15e7634be97f2f1" exitCode=0
Sep 05 02:18:06.224104 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:06.223728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"34a9133e817a564e573345fb80126e7d804dc724b9295958c15e7634be97f2f1"}
Sep 05 02:18:06.965673 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:06.965648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:06.965912 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:06.965764    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:06.965912 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:06.965774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:06.966066 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:06.966037    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:07.965109 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:07.965041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:07.965492 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:07.965169    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:08.231041 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:08.230959    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="7efe67701ce81f3b4e750090b746f458128de265f3c4d707c0583e830343cf6e" exitCode=0
Sep 05 02:18:08.231041 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:08.231034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"7efe67701ce81f3b4e750090b746f458128de265f3c4d707c0583e830343cf6e"}
Sep 05 02:18:08.964815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:08.964782    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:08.965024 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:08.964923    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:08.965024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:08.964989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:08.965213 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:08.965181    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:09.964717 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:09.964686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:09.964888 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:09.964806    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:10.479436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:10.479402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:10.479897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:10.479458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:10.479897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.479589    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:10.479897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.479625    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:10.479897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.479656    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret podName:fda2fbf9-4539-4dec-afee-46b0f5635648 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.479637374 +0000 UTC m=+66.118849163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret") pod "global-pull-secret-syncer-8zl8r" (UID: "fda2fbf9-4539-4dec-afee-46b0f5635648") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:10.479897 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.479677    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.479667428 +0000 UTC m=+66.118879220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:10.681324 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:10.681293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:10.681481 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.681402    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:10.681481 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.681415    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:10.681481 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.681424    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tqvbw for pod openshift-network-diagnostics/network-check-target-lb7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:10.681481 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.681467    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw podName:e66d0bea-4ac7-4181-a160-1f1b3b347c13 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.681455445 +0000 UTC m=+66.320667223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tqvbw" (UniqueName: "kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw") pod "network-check-target-lb7h9" (UID: "e66d0bea-4ac7-4181-a160-1f1b3b347c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:10.964853 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:10.964771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:10.965111 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:10.964779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:10.965111 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.964905    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zl8r" podUID="fda2fbf9-4539-4dec-afee-46b0f5635648"
Sep 05 02:18:10.965111 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:10.964968    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mj9x2" podUID="fcc83524-5eb0-4089-8aae-08ff8edd38c6"
Sep 05 02:18:11.965072 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:11.965034    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:11.965526 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:11.965161    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lb7h9" podUID="e66d0bea-4ac7-4181-a160-1f1b3b347c13"
Sep 05 02:18:12.653098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.653072    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-13.ec2.internal" event="NodeReady"
Sep 05 02:18:12.653230 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.653191    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:12.699817 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.699791    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67bd46df57-gpl5k"]
Sep 05 02:18:12.702945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.702924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.706278 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.706256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:18:12.706505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.706479    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:18:12.706584 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.706538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:18:12.706584 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.706557    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7th2q\""
Sep 05 02:18:12.714817 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.714618    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:18:12.729854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.729834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67bd46df57-gpl5k"]
Sep 05 02:18:12.730917 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.730897    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qf7hv"]
Sep 05 02:18:12.734059 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.734034    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jd8"]
Sep 05 02:18:12.734153 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.734136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.737067 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.737049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:12.737211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.737191    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:12.737306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.737256    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:12.737306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.737278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2swhn\""
Sep 05 02:18:12.739967 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.739943    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.740247 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.740232    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:12.740294 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.740234    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dnzd9\""
Sep 05 02:18:12.740847 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.740834    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.747608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.747585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jd8"]
Sep 05 02:18:12.748506 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.748488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qf7hv"]
Sep 05 02:18:12.796437 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqf7\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.796799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.796678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0078607-a4d5-4899-825f-ea220eb8f185-cert\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:12.897359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp6g\" (UniqueName: \"kubernetes.io/projected/f0078607-a4d5-4899-825f-ea220eb8f185-kube-api-access-zjp6g\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:12.897359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-tmp-dir\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.897480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.897480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqf7\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm485\" (UniqueName: \"kubernetes.io/projected/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-kube-api-access-rm485\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:12.897519    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:12.897537    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-config-volume\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.897687 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:12.897601    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.397582251 +0000 UTC m=+37.036794042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:12.897950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.897902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.898065 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.898046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.898116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.898106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.901381 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.901357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.901477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.901357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.906338 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.906319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.906820 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.906805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqf7\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:12.964965 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.964943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:12.965047 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.964943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:12.969039 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.969021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:12.969337 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.969056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:12.969337 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.969314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptsx2\""
Sep 05 02:18:12.998396 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0078607-a4d5-4899-825f-ea220eb8f185-cert\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:12.998472 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp6g\" (UniqueName: \"kubernetes.io/projected/f0078607-a4d5-4899-825f-ea220eb8f185-kube-api-access-zjp6g\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:12.998472 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-tmp-dir\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.998472 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.998472 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm485\" (UniqueName: \"kubernetes.io/projected/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-kube-api-access-rm485\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.998619 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-config-volume\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.998619 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:12.998536    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:12.998619 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:12.998600    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.49858133 +0000 UTC m=+37.137793112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:12.998797 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-tmp-dir\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:12.998967 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:12.998952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-config-volume\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:13.000413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.000396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0078607-a4d5-4899-825f-ea220eb8f185-cert\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:13.009289 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.009264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp6g\" (UniqueName: \"kubernetes.io/projected/f0078607-a4d5-4899-825f-ea220eb8f185-kube-api-access-zjp6g\") pod \"ingress-canary-v5jd8\" (UID: \"f0078607-a4d5-4899-825f-ea220eb8f185\") " pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:13.009597 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.009577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm485\" (UniqueName: \"kubernetes.io/projected/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-kube-api-access-rm485\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:13.054884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.054837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5jd8"
Sep 05 02:18:13.244854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.244825    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="082e337918f0e51a56ee065ed566f44653f5be4f3ee33bd0b674acdef03dc14b" exitCode=0
Sep 05 02:18:13.244972 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.244886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"082e337918f0e51a56ee065ed566f44653f5be4f3ee33bd0b674acdef03dc14b"}
Sep 05 02:18:13.332439 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.332411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jd8"]
Sep 05 02:18:13.335997 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:18:13.335954    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0078607_a4d5_4899_825f_ea220eb8f185.slice/crio-6ac5f4ee68cc948358d81e0acff2d207cb568ccf5f07d1d047f839ef6dc234dd WatchSource:0}: Error finding container 6ac5f4ee68cc948358d81e0acff2d207cb568ccf5f07d1d047f839ef6dc234dd: Status 404 returned error can't find the container with id 6ac5f4ee68cc948358d81e0acff2d207cb568ccf5f07d1d047f839ef6dc234dd
Sep 05 02:18:13.402038 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.402015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:13.402175 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:13.402153    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:13.402239 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:13.402178    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:13.402295 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:13.402239    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:14.402217692 +0000 UTC m=+38.041429490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:13.503101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.503078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:13.503230 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:13.503212    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.503280 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:13.503271    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:14.50325229 +0000 UTC m=+38.142464085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.964839 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.964809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:13.967728 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.967703    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fnrtr\""
Sep 05 02:18:13.967866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.967768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:13.968812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:13.968792    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:14.248484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.248402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5jd8" event={"ID":"f0078607-a4d5-4899-825f-ea220eb8f185","Type":"ContainerStarted","Data":"6ac5f4ee68cc948358d81e0acff2d207cb568ccf5f07d1d047f839ef6dc234dd"}
Sep 05 02:18:14.251999 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.251972    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e624dab-2c86-455c-a2f0-78752f63e187" containerID="80735959317d126ebc495f2c1483f7f1b872b63e6d0bdb94774301bb52e2514f" exitCode=0
Sep 05 02:18:14.252116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.252029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerDied","Data":"80735959317d126ebc495f2c1483f7f1b872b63e6d0bdb94774301bb52e2514f"}
Sep 05 02:18:14.344771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.344656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:14.409872 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.409838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:14.410186 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:14.410169    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:14.410252 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:14.410190    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:14.410305 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:14.410255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:16.410237855 +0000 UTC m=+40.049449636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:14.462579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.462537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:14.510867 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.510764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:14.510867 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:14.510852    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:14.511055 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:14.510928    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:16.510905282 +0000 UTC m=+40.150117062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:14.551510 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:14.551456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:15.259122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.259094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" event={"ID":"5e624dab-2c86-455c-a2f0-78752f63e187","Type":"ContainerStarted","Data":"a5e1bd5814474f05a372c1185e8c338d379de611f3f65a47114af16031149b50"}
Sep 05 02:18:15.283029 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.282978    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f2hw6" podStartSLOduration=8.932817527 podStartE2EDuration="38.282958768s" totalImagesPullingTime="29.350141241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:15.281092864 +0000 UTC m=+38.920304694" watchObservedRunningTime="2026-09-05 02:18:15.282958768 +0000 UTC m=+38.922170622"
Sep 05 02:18:15.444489 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.444451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:15.521217 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.521189    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:15.916094 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.916056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:15.973911 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:15.973874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:16.263391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:16.263352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5jd8" event={"ID":"f0078607-a4d5-4899-825f-ea220eb8f185","Type":"ContainerStarted","Data":"9df4e0877eded326349330e9a04d1c27447731fee81dc9c969c3aeba6ab10848"}
Sep 05 02:18:16.281906 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:16.281860    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v5jd8" podStartSLOduration=1.824174024 podStartE2EDuration="4.281844618s" totalImagesPullingTime="2.457670594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:16.28009558 +0000 UTC m=+39.919307397" watchObservedRunningTime="2026-09-05 02:18:16.281844618 +0000 UTC m=+39.921056407"
Sep 05 02:18:16.424424 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:16.424297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:16.424424 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:16.424420    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:16.424627 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:16.424435    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:16.424627 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:16.424492    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.424478838 +0000 UTC m=+44.063690617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:16.524814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:16.524736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:16.524904 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:16.524851    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:16.524904 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:16.524901    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.524886498 +0000 UTC m=+44.164098276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:20.449269 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:20.449236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:20.449611 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:20.449341    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:20.449611 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:20.449353    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:20.449611 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:20.449400    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:28.449386924 +0000 UTC m=+52.088598701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:20.549976 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:20.549946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:20.550095 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:20.550077    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:20.550136 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:20.550126    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:28.550114235 +0000 UTC m=+52.189326014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:28.504705 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:28.504665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:28.505152 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:28.504829    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:28.505152 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:28.504847    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:28.505152 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:28.504914    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:18:44.50489725 +0000 UTC m=+68.144109028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:28.605752 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:28.605708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:28.605888 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:28.605861    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:28.605930 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:28.605920    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:44.60590186 +0000 UTC m=+68.245113653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:37.236483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:37.236455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncjl"
Sep 05 02:18:42.506304 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.506256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:42.506800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.506319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:18:42.509221 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.509199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:42.509309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.509219    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:42.516494 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:42.516472    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:18:42.516590 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:42.516521    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs podName:fcc83524-5eb0-4089-8aae-08ff8edd38c6 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:46.516506163 +0000 UTC m=+130.155717941 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs") pod "network-metrics-daemon-mj9x2" (UID: "fcc83524-5eb0-4089-8aae-08ff8edd38c6") : secret "metrics-daemon-secret" not found
Sep 05 02:18:42.519377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.519351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda2fbf9-4539-4dec-afee-46b0f5635648-original-pull-secret\") pod \"global-pull-secret-syncer-8zl8r\" (UID: \"fda2fbf9-4539-4dec-afee-46b0f5635648\") " pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:42.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.678858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zl8r"
Sep 05 02:18:42.707671 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.707646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:42.710589 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.710565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:42.721335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.721316    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:42.733525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.733472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvbw\" (UniqueName: \"kubernetes.io/projected/e66d0bea-4ac7-4181-a160-1f1b3b347c13-kube-api-access-tqvbw\") pod \"network-check-target-lb7h9\" (UID: \"e66d0bea-4ac7-4181-a160-1f1b3b347c13\") " pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:42.778100 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.778047    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fnrtr\""
Sep 05 02:18:42.786183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.786165    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:42.952297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:42.952270    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zl8r"]
Sep 05 02:18:42.955126 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:18:42.955103    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda2fbf9_4539_4dec_afee_46b0f5635648.slice/crio-66a210d26ef3852abbeaaea81ca80ba800f3defda881e628232f95451489b589 WatchSource:0}: Error finding container 66a210d26ef3852abbeaaea81ca80ba800f3defda881e628232f95451489b589: Status 404 returned error can't find the container with id 66a210d26ef3852abbeaaea81ca80ba800f3defda881e628232f95451489b589
Sep 05 02:18:43.066831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:43.066768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lb7h9"]
Sep 05 02:18:43.070448 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:18:43.070426    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66d0bea_4ac7_4181_a160_1f1b3b347c13.slice/crio-3120b31b6e0223d9f39c6c62f5b6f98a14115828d851dd1a4fca9bedf8780634 WatchSource:0}: Error finding container 3120b31b6e0223d9f39c6c62f5b6f98a14115828d851dd1a4fca9bedf8780634: Status 404 returned error can't find the container with id 3120b31b6e0223d9f39c6c62f5b6f98a14115828d851dd1a4fca9bedf8780634
Sep 05 02:18:43.326700 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:43.326623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lb7h9" event={"ID":"e66d0bea-4ac7-4181-a160-1f1b3b347c13","Type":"ContainerStarted","Data":"3120b31b6e0223d9f39c6c62f5b6f98a14115828d851dd1a4fca9bedf8780634"}
Sep 05 02:18:43.327681 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:43.327650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8zl8r" event={"ID":"fda2fbf9-4539-4dec-afee-46b0f5635648","Type":"ContainerStarted","Data":"66a210d26ef3852abbeaaea81ca80ba800f3defda881e628232f95451489b589"}
Sep 05 02:18:44.521111 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:44.521046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:18:44.521585 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:44.521178    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:44.521585 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:44.521203    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:18:44.521585 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:44.521278    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.521256032 +0000 UTC m=+100.160467830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:18:44.621697 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:44.621660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:18:44.621890 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:44.621817    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:44.621955 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:18:44.621903    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls podName:17eded11-bbfc-4a5e-a7d7-f22392d7a5f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.621881344 +0000 UTC m=+100.261093121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls") pod "dns-default-qf7hv" (UID: "17eded11-bbfc-4a5e-a7d7-f22392d7a5f8") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:48.342559 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:48.342508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lb7h9" event={"ID":"e66d0bea-4ac7-4181-a160-1f1b3b347c13","Type":"ContainerStarted","Data":"17f5cf305d0c5fc89daa909d60bbd1b3fa31b074d3219bc8bcab0a1d73b73d4c"}
Sep 05 02:18:48.343051 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:48.342581    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:18:48.344146 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:48.344118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8zl8r" event={"ID":"fda2fbf9-4539-4dec-afee-46b0f5635648","Type":"ContainerStarted","Data":"16d27f3e849e99e72fba17947b3b838841dc45df01bd40d9003b3e91874fdc85"}
Sep 05 02:18:48.362579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:48.362531    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lb7h9" podStartSLOduration=66.884786088 podStartE2EDuration="1m11.362518196s" totalImagesPullingTime="4.477732108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:48.361923707 +0000 UTC m=+72.001135522" watchObservedRunningTime="2026-09-05 02:18:48.362518196 +0000 UTC m=+72.001729997"
Sep 05 02:18:48.379527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:18:48.379488    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8zl8r" podStartSLOduration=66.774126942 podStartE2EDuration="1m11.379473778s" totalImagesPullingTime="4.605346836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:48.379406857 +0000 UTC m=+72.018618675" watchObservedRunningTime="2026-09-05 02:18:48.379473778 +0000 UTC m=+72.018685578"
Sep 05 02:19:03.515104 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.514959    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"]
Sep 05 02:19:03.519588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.519562    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.525873 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.525637    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"]
Sep 05 02:19:03.525978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.525953    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 05 02:19:03.526091 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.525953    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.526091 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.525952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 05 02:19:03.526182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.526091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ttcrk\""
Sep 05 02:19:03.527197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.527180    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.528622 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.528607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.535815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.535797    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 05 02:19:03.535904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.535813    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 05 02:19:03.535904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.535826    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.535904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.535853    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r55m5\""
Sep 05 02:19:03.535904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.535883    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.545100 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.545079    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"]
Sep 05 02:19:03.545850 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.545830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"]
Sep 05 02:19:03.642717 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ab1399-a71b-48c6-83d3-0b1d33539384-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.642832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvv9\" (UniqueName: \"kubernetes.io/projected/77ab1399-a71b-48c6-83d3-0b1d33539384-kube-api-access-tdvv9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.642832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/77ab1399-a71b-48c6-83d3-0b1d33539384-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.642832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d9ece26-4f8b-4ea7-9525-fb474583095e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.642832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl25n\" (UniqueName: \"kubernetes.io/projected/3d9ece26-4f8b-4ea7-9525-fb474583095e-kube-api-access-cl25n\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.642962 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.642838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.706030 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.706006    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-hp7jp"]
Sep 05 02:19:03.708831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.708814    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.712953 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.712934    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.713045 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.712960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 05 02:19:03.713126 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.713109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ks579\""
Sep 05 02:19:03.713719 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.713693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.713841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.713793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 05 02:19:03.743633 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp4k5\" (UniqueName: \"kubernetes.io/projected/21a849dc-df64-4332-803b-45cdb526a487-kube-api-access-sp4k5\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.743724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-tmp\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.743724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/77ab1399-a71b-48c6-83d3-0b1d33539384-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.743863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d9ece26-4f8b-4ea7-9525-fb474583095e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.743863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl25n\" (UniqueName: \"kubernetes.io/projected/3d9ece26-4f8b-4ea7-9525-fb474583095e-kube-api-access-cl25n\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.743863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-snapshots\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.743974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a849dc-df64-4332-803b-45cdb526a487-serving-cert\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:03.743996    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:03.744019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.744008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ab1399-a71b-48c6-83d3-0b1d33539384-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.744301 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.744040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvv9\" (UniqueName: \"kubernetes.io/projected/77ab1399-a71b-48c6-83d3-0b1d33539384-kube-api-access-tdvv9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.744301 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:03.744082    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:04.244044972 +0000 UTC m=+87.883256756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:03.744423 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.744302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/77ab1399-a71b-48c6-83d3-0b1d33539384-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.744423 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.744406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d9ece26-4f8b-4ea7-9525-fb474583095e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.744945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.744928    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-hp7jp"]
Sep 05 02:19:03.746389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.746369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ab1399-a71b-48c6-83d3-0b1d33539384-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.762708 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.759930    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 05 02:19:03.787027 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.786980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl25n\" (UniqueName: \"kubernetes.io/projected/3d9ece26-4f8b-4ea7-9525-fb474583095e-kube-api-access-cl25n\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:03.787288 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.787267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvv9\" (UniqueName: \"kubernetes.io/projected/77ab1399-a71b-48c6-83d3-0b1d33539384-kube-api-access-tdvv9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vltzw\" (UID: \"77ab1399-a71b-48c6-83d3-0b1d33539384\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.828197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.828176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"
Sep 05 02:19:03.844548 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp4k5\" (UniqueName: \"kubernetes.io/projected/21a849dc-df64-4332-803b-45cdb526a487-kube-api-access-sp4k5\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-tmp\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-snapshots\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844801 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844801 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a849dc-df64-4332-803b-45cdb526a487-serving-cert\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.844997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.844978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-tmp\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.845325 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.845295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.845420 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.845327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21a849dc-df64-4332-803b-45cdb526a487-snapshots\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.845487 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.845446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a849dc-df64-4332-803b-45cdb526a487-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.846729 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.846709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a849dc-df64-4332-803b-45cdb526a487-serving-cert\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:03.853904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:03.853880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp4k5\" (UniqueName: \"kubernetes.io/projected/21a849dc-df64-4332-803b-45cdb526a487-kube-api-access-sp4k5\") pod \"insights-operator-76bcd6ccf5-hp7jp\" (UID: \"21a849dc-df64-4332-803b-45cdb526a487\") " pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:04.017373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.017345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp"
Sep 05 02:19:04.102797 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.102769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw"]
Sep 05 02:19:04.105414 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:04.105392    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ab1399_a71b_48c6_83d3_0b1d33539384.slice/crio-96ed3640278fa3234ce00298f77b4d3fccf6f969b7e7d6202f53db0b09bb82c2 WatchSource:0}: Error finding container 96ed3640278fa3234ce00298f77b4d3fccf6f969b7e7d6202f53db0b09bb82c2: Status 404 returned error can't find the container with id 96ed3640278fa3234ce00298f77b4d3fccf6f969b7e7d6202f53db0b09bb82c2
Sep 05 02:19:04.247978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.247953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:04.248079 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:04.248067    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:04.248120 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:04.248113    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.248100972 +0000 UTC m=+88.887312750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:04.283733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.283709    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-hp7jp"]
Sep 05 02:19:04.286334 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:04.286313    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a849dc_df64_4332_803b_45cdb526a487.slice/crio-ab2dd5629c89a8052aebd182dc853c5d353bde69afe9bacda472c0e23613324b WatchSource:0}: Error finding container ab2dd5629c89a8052aebd182dc853c5d353bde69afe9bacda472c0e23613324b: Status 404 returned error can't find the container with id ab2dd5629c89a8052aebd182dc853c5d353bde69afe9bacda472c0e23613324b
Sep 05 02:19:04.380765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.380689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp" event={"ID":"21a849dc-df64-4332-803b-45cdb526a487","Type":"ContainerStarted","Data":"ab2dd5629c89a8052aebd182dc853c5d353bde69afe9bacda472c0e23613324b"}
Sep 05 02:19:04.381766 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:04.381729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw" event={"ID":"77ab1399-a71b-48c6-83d3-0b1d33539384","Type":"ContainerStarted","Data":"96ed3640278fa3234ce00298f77b4d3fccf6f969b7e7d6202f53db0b09bb82c2"}
Sep 05 02:19:05.257554 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:05.257520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:05.257994 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:05.257691    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:05.257994 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:05.257786    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.257764855 +0000 UTC m=+90.896976652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:06.389344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:06.389306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp" event={"ID":"21a849dc-df64-4332-803b-45cdb526a487","Type":"ContainerStarted","Data":"1b4b75536faa826e3b29313407edff7568b22e1f420d6880357281137da60aaa"}
Sep 05 02:19:06.410675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:06.410623    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp" podStartSLOduration=1.49815821 podStartE2EDuration="3.410607493s" totalImagesPullingTime="1.912449283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:06.40949674 +0000 UTC m=+90.048708566" watchObservedRunningTime="2026-09-05 02:19:06.410607493 +0000 UTC m=+90.049819292"
Sep 05 02:19:07.274844 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:07.274800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:07.275083 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:07.274931    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:07.275083 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:07.274997    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:11.274978472 +0000 UTC m=+94.914190266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:08.399917 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.399875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw" event={"ID":"77ab1399-a71b-48c6-83d3-0b1d33539384","Type":"ContainerStarted","Data":"ca09d9dcf121d7c82994165d0f4077a5151b89e0e378d549b055ce4e93115fe8"}
Sep 05 02:19:08.426962 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.426915    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw" podStartSLOduration=2.111933832 podStartE2EDuration="5.42689976s" totalImagesPullingTime="3.314965928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:08.426115045 +0000 UTC m=+92.065326846" watchObservedRunningTime="2026-09-05 02:19:08.42689976 +0000 UTC m=+92.066111559"
Sep 05 02:19:08.980795 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.980762    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mnk4"]
Sep 05 02:19:08.983870 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.983848    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:08.986521 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.986500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:19:08.986631 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.986523    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-twl8z\""
Sep 05 02:19:08.986831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.986813    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:19:08.997250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:08.997230    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mnk4"]
Sep 05 02:19:09.088156 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.088121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.088156 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.088157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-api-access-9r2lv\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.088342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.088198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.088342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.088290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-crio-socket\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.088342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.088320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-data-volume\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189599 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-data-volume\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-api-access-9r2lv\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189784 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.189700    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.189960 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189960 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-crio-socket\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189960 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.189897    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:09.689872684 +0000 UTC m=+93.329084475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.189960 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-crio-socket\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.189960 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.189933    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.190130 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.189966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-data-volume\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.190130 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.189983    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:09.689975131 +0000 UTC m=+93.329186908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.201932 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.201911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-api-access-9r2lv\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.694116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.694070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.694492 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:09.694134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:09.694492 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.694182    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.694492 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.694239    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.694492 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.694247    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:10.694229668 +0000 UTC m=+94.333441446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.694492 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:09.694279    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:10.694267789 +0000 UTC m=+94.333479567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:10.064790 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.064695    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4m8g"]
Sep 05 02:19:10.067545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.067530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.070445 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.070423    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 05 02:19:10.070765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.070733    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 05 02:19:10.070889 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.070871    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 05 02:19:10.071819 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.071800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rv69h\""
Sep 05 02:19:10.071970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.071953    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 05 02:19:10.078495 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.078474    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4m8g"]
Sep 05 02:19:10.198267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.198242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-cabundle\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.198390 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.198285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldh7\" (UniqueName: \"kubernetes.io/projected/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-kube-api-access-kldh7\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.198390 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.198304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-key\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.299418 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.299393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldh7\" (UniqueName: \"kubernetes.io/projected/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-kube-api-access-kldh7\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.299517 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.299423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-key\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.299679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.299660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-cabundle\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.300221 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.300202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-cabundle\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.301823 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.301806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-signing-key\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.308916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.308888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldh7\" (UniqueName: \"kubernetes.io/projected/ace8c24f-5aaf-4bab-a6c3-c09890269bcf-kube-api-access-kldh7\") pod \"service-ca-75d99c484-d4m8g\" (UID: \"ace8c24f-5aaf-4bab-a6c3-c09890269bcf\") " pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.377065 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.377017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d4m8g"
Sep 05 02:19:10.655829 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.655798    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4m8g"]
Sep 05 02:19:10.658564 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:10.658524    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace8c24f_5aaf_4bab_a6c3_c09890269bcf.slice/crio-35832ece1729455080b2c9af068e1976a6e75c5c2737087a2d40a77453bc0694 WatchSource:0}: Error finding container 35832ece1729455080b2c9af068e1976a6e75c5c2737087a2d40a77453bc0694: Status 404 returned error can't find the container with id 35832ece1729455080b2c9af068e1976a6e75c5c2737087a2d40a77453bc0694
Sep 05 02:19:10.701772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.701727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:10.702080 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:10.701842    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:10.702080 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:10.701896    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:12.701881696 +0000 UTC m=+96.341093479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:10.702080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:10.701892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:10.702080 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:10.701961    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:10.702080 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:10.702006    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:12.701995522 +0000 UTC m=+96.341207304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:11.305044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:11.305007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:11.305212 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:11.305154    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:11.305252 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:11.305226    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:19.305207054 +0000 UTC m=+102.944418832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:11.413871 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:11.413826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d4m8g" event={"ID":"ace8c24f-5aaf-4bab-a6c3-c09890269bcf","Type":"ContainerStarted","Data":"35832ece1729455080b2c9af068e1976a6e75c5c2737087a2d40a77453bc0694"}
Sep 05 02:19:11.744897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:11.744872    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r92cr_fb62105b-3282-459d-a241-fb10b3ae0191/dns-node-resolver/0.log"
Sep 05 02:19:12.418549 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:12.418515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d4m8g" event={"ID":"ace8c24f-5aaf-4bab-a6c3-c09890269bcf","Type":"ContainerStarted","Data":"188f7a4d71e8d61cd1dec397db06ce294eb620463e2436960ce353162f57a778"}
Sep 05 02:19:12.450701 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:12.450656    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-d4m8g" podStartSLOduration=0.867472482 podStartE2EDuration="2.450642026s" totalImagesPullingTime="1.583169544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:12.450590292 +0000 UTC m=+96.089802091" watchObservedRunningTime="2026-09-05 02:19:12.450642026 +0000 UTC m=+96.089853825"
Sep 05 02:19:12.715326 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:12.715294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:12.715465 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:12.715354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:12.715465 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:12.715400    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:12.715465 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:12.715455    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.715439252 +0000 UTC m=+100.354651030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:12.715593 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:12.715469    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:12.715593 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:12.715505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.715494837 +0000 UTC m=+100.354706615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:12.740699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:12.740674    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7n8lq_b29d312f-67d2-4530-bcfd-b6e7f1dada9c/node-ca/0.log"
Sep 05 02:19:13.541856 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:13.541812    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5jd8_f0078607-a4d5-4899-825f-ea220eb8f185/serve-healthcheck-canary/0.log"
Sep 05 02:19:14.342441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:14.342410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vltzw_77ab1399-a71b-48c6-83d3-0b1d33539384/kube-storage-version-migrator-operator/0.log"
Sep 05 02:19:16.545621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.545583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") pod \"image-registry-67bd46df57-gpl5k\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") " pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:19:16.546052 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.545734    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:16.546052 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.545771    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd46df57-gpl5k: secret "image-registry-tls" not found
Sep 05 02:19:16.546052 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.545829    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls podName:a57eb46c-4ae4-4b55-9d65-57c801029bae nodeName:}" failed. No retries permitted until 2026-09-05 02:20:20.545810319 +0000 UTC m=+164.185022101 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls") pod "image-registry-67bd46df57-gpl5k" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae") : secret "image-registry-tls" not found
Sep 05 02:19:16.646122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.646091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:19:16.648488 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.648466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17eded11-bbfc-4a5e-a7d7-f22392d7a5f8-metrics-tls\") pod \"dns-default-qf7hv\" (UID: \"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8\") " pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:19:16.657072 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.657042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2swhn\""
Sep 05 02:19:16.660603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.660583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:19:16.747386 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.747352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:16.747529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.747490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:16.747577 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.747533    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:16.747577 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.747552    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:16.747646 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.747613    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.747595388 +0000 UTC m=+108.386807186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:16.747646 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:16.747629    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm podName:6cbf3b6f-e776-4bd5-8d0d-354cd5314481 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.747622762 +0000 UTC m=+108.386834540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mnk4" (UID: "6cbf3b6f-e776-4bd5-8d0d-354cd5314481") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:16.969685 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:16.969655    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qf7hv"]
Sep 05 02:19:16.972835 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:16.972802    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eded11_bbfc_4a5e_a7d7_f22392d7a5f8.slice/crio-66e30127256c8cd984573e0787ebd81a4873ccc4d75c4022becf2621eb983c72 WatchSource:0}: Error finding container 66e30127256c8cd984573e0787ebd81a4873ccc4d75c4022becf2621eb983c72: Status 404 returned error can't find the container with id 66e30127256c8cd984573e0787ebd81a4873ccc4d75c4022becf2621eb983c72
Sep 05 02:19:17.435089 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:17.435057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf7hv" event={"ID":"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8","Type":"ContainerStarted","Data":"66e30127256c8cd984573e0787ebd81a4873ccc4d75c4022becf2621eb983c72"}
Sep 05 02:19:19.348287 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.348263    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lb7h9"
Sep 05 02:19:19.368883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.368858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:19.369037 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:19.369008    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:19.369100 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:19.369080    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls podName:3d9ece26-4f8b-4ea7-9525-fb474583095e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:35.36905546 +0000 UTC m=+119.008267260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6k4zm" (UID: "3d9ece26-4f8b-4ea7-9525-fb474583095e") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:19.442760 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.442710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf7hv" event={"ID":"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8","Type":"ContainerStarted","Data":"d43c454a53edc4410d81de77821fb2d2772f4979b4edfc895994056aade7a0d8"}
Sep 05 02:19:19.442941 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.442771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf7hv" event={"ID":"17eded11-bbfc-4a5e-a7d7-f22392d7a5f8","Type":"ContainerStarted","Data":"27e0e5852cb146ba36c2f70aeff6a5992c02d22a1d7d0e5e27b6009f242113c0"}
Sep 05 02:19:19.442941 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.442860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:19:19.461019 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:19.460981    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qf7hv" podStartSLOduration=65.30672972 podStartE2EDuration="1m7.460968771s" totalImagesPullingTime="2.154239051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:19.460419852 +0000 UTC m=+103.099631670" watchObservedRunningTime="2026-09-05 02:19:19.460968771 +0000 UTC m=+103.100180571"
Sep 05 02:19:24.808233 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:24.808198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:24.808602 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:24.808255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:24.808729 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:24.808705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:24.810474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:24.810454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cbf3b6f-e776-4bd5-8d0d-354cd5314481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mnk4\" (UID: \"6cbf3b6f-e776-4bd5-8d0d-354cd5314481\") " pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:24.892751 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:24.892716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mnk4"
Sep 05 02:19:25.190144 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:25.190095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mnk4"]
Sep 05 02:19:25.194290 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:25.194264    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbf3b6f_e776_4bd5_8d0d_354cd5314481.slice/crio-8638ec813bd55c6dca9896c36db76321eabb3c0a06199e29b9515622cc90fa41 WatchSource:0}: Error finding container 8638ec813bd55c6dca9896c36db76321eabb3c0a06199e29b9515622cc90fa41: Status 404 returned error can't find the container with id 8638ec813bd55c6dca9896c36db76321eabb3c0a06199e29b9515622cc90fa41
Sep 05 02:19:25.462631 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:25.462550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mnk4" event={"ID":"6cbf3b6f-e776-4bd5-8d0d-354cd5314481","Type":"ContainerStarted","Data":"827f891a010d360094290d3e9294118f9a1808d478310259ed281d3c984f0319"}
Sep 05 02:19:25.462631 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:25.462605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mnk4" event={"ID":"6cbf3b6f-e776-4bd5-8d0d-354cd5314481","Type":"ContainerStarted","Data":"8638ec813bd55c6dca9896c36db76321eabb3c0a06199e29b9515622cc90fa41"}
Sep 05 02:19:27.469809 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:27.469727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mnk4" event={"ID":"6cbf3b6f-e776-4bd5-8d0d-354cd5314481","Type":"ContainerStarted","Data":"2ed4fa0c9e3e8c45128da4627ae2e9ab9ca9eb0fe736f29fd1974bd93ea60cf2"}
Sep 05 02:19:28.474234 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:28.474193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mnk4" event={"ID":"6cbf3b6f-e776-4bd5-8d0d-354cd5314481","Type":"ContainerStarted","Data":"f13315e51fdd350178525ee71baedc169c535310956aa0c2f20513e6517d11c3"}
Sep 05 02:19:28.493174 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:28.493133    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mnk4" podStartSLOduration=17.39088058 podStartE2EDuration="20.49311905s" totalImagesPullingTime="3.10223847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:28.491850071 +0000 UTC m=+112.131061892" watchObservedRunningTime="2026-09-05 02:19:28.49311905 +0000 UTC m=+112.132330849"
Sep 05 02:19:29.448042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:29.448003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qf7hv"
Sep 05 02:19:30.757669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.757637    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"]
Sep 05 02:19:30.760684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.760669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.767789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.767769    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:30.768952 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.768934    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:30.769051 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.768951    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 05 02:19:30.769051 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.769025    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:30.800845 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.800821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"]
Sep 05 02:19:30.852226 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.852148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2385126f-c8c5-41ea-8f35-674c3f478baf-klusterlet-config\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.852226 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.852196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2385126f-c8c5-41ea-8f35-674c3f478baf-tmp\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.852385 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.852268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bd5b\" (UniqueName: \"kubernetes.io/projected/2385126f-c8c5-41ea-8f35-674c3f478baf-kube-api-access-4bd5b\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.952816 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.952788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bd5b\" (UniqueName: \"kubernetes.io/projected/2385126f-c8c5-41ea-8f35-674c3f478baf-kube-api-access-4bd5b\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.952954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.952854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2385126f-c8c5-41ea-8f35-674c3f478baf-klusterlet-config\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.952954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.952878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2385126f-c8c5-41ea-8f35-674c3f478baf-tmp\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.953236 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.953215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2385126f-c8c5-41ea-8f35-674c3f478baf-tmp\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:30.955218 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:30.955200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2385126f-c8c5-41ea-8f35-674c3f478baf-klusterlet-config\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:31.018585 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:31.018523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bd5b\" (UniqueName: \"kubernetes.io/projected/2385126f-c8c5-41ea-8f35-674c3f478baf-kube-api-access-4bd5b\") pod \"klusterlet-addon-workmgr-569ff46c46-bm4fx\" (UID: \"2385126f-c8c5-41ea-8f35-674c3f478baf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:31.069208 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:31.069187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:31.378483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:31.378459    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"]
Sep 05 02:19:31.380955 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:31.380926    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2385126f_c8c5_41ea_8f35_674c3f478baf.slice/crio-f22d3996c11253e1eb2e961e9fff0777973dcd3bad9892422a543c661a293255 WatchSource:0}: Error finding container f22d3996c11253e1eb2e961e9fff0777973dcd3bad9892422a543c661a293255: Status 404 returned error can't find the container with id f22d3996c11253e1eb2e961e9fff0777973dcd3bad9892422a543c661a293255
Sep 05 02:19:31.485445 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:31.485417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx" event={"ID":"2385126f-c8c5-41ea-8f35-674c3f478baf","Type":"ContainerStarted","Data":"f22d3996c11253e1eb2e961e9fff0777973dcd3bad9892422a543c661a293255"}
Sep 05 02:19:35.386790 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.386757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:35.389252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.389217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d9ece26-4f8b-4ea7-9525-fb474583095e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6k4zm\" (UID: \"3d9ece26-4f8b-4ea7-9525-fb474583095e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:35.501526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.501481    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx" event={"ID":"2385126f-c8c5-41ea-8f35-674c3f478baf","Type":"ContainerStarted","Data":"8ce0d5136e88352b3b7d82fab868df5b716dc6459fdf0f7d6ef3c070e82c3643"}
Sep 05 02:19:35.501707 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.501683    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:35.503243 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.503220    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx"
Sep 05 02:19:35.517864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.517821    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-569ff46c46-bm4fx" podStartSLOduration=1.970674542 podStartE2EDuration="5.517809656s" totalImagesPullingTime="3.547135114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:35.517544571 +0000 UTC m=+119.156756404" watchObservedRunningTime="2026-09-05 02:19:35.517809656 +0000 UTC m=+119.157021457"
Sep 05 02:19:35.636636 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.636604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"
Sep 05 02:19:35.907561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:35.907491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm"]
Sep 05 02:19:35.910550 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:35.910518    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ece26_4f8b_4ea7_9525_fb474583095e.slice/crio-7e9ce94f303307e2f457695ea857c09b3ea6b51197734573929615b9252834a5 WatchSource:0}: Error finding container 7e9ce94f303307e2f457695ea857c09b3ea6b51197734573929615b9252834a5: Status 404 returned error can't find the container with id 7e9ce94f303307e2f457695ea857c09b3ea6b51197734573929615b9252834a5
Sep 05 02:19:36.507021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:36.506976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm" event={"ID":"3d9ece26-4f8b-4ea7-9525-fb474583095e","Type":"ContainerStarted","Data":"7e9ce94f303307e2f457695ea857c09b3ea6b51197734573929615b9252834a5"}
Sep 05 02:19:37.511468 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.511441    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm" event={"ID":"3d9ece26-4f8b-4ea7-9525-fb474583095e","Type":"ContainerStarted","Data":"ef26fd32c4f545a4a1883b8328f86da8c495f5c8133e1cd43d2adbc76c24f941"}
Sep 05 02:19:37.530896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.530842    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6k4zm" podStartSLOduration=32.995056741 podStartE2EDuration="34.530803864s" totalImagesPullingTime="1.535747123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:37.52991937 +0000 UTC m=+121.169131171" watchObservedRunningTime="2026-09-05 02:19:37.530803864 +0000 UTC m=+121.170015664"
Sep 05 02:19:37.961327 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.961291    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"]
Sep 05 02:19:37.964724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.964705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:37.968510 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.968486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 05 02:19:37.968613 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.968523    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cxnxt\""
Sep 05 02:19:37.980955 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:37.980933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"]
Sep 05 02:19:38.108972 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:38.108943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8qvbm\" (UID: \"8bceb5c2-cca1-49e0-91f0-9739f956bf2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:38.210135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:38.210109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8qvbm\" (UID: \"8bceb5c2-cca1-49e0-91f0-9739f956bf2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:38.210299 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:38.210278    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:38.210372 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:38.210361    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates podName:8bceb5c2-cca1-49e0-91f0-9739f956bf2a nodeName:}" failed. No retries permitted until 2026-09-05 02:19:38.710340024 +0000 UTC m=+122.349551816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-8qvbm" (UID: "8bceb5c2-cca1-49e0-91f0-9739f956bf2a") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:38.714188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:38.714148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8qvbm\" (UID: \"8bceb5c2-cca1-49e0-91f0-9739f956bf2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:38.716461 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:38.716435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bceb5c2-cca1-49e0-91f0-9739f956bf2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8qvbm\" (UID: \"8bceb5c2-cca1-49e0-91f0-9739f956bf2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:38.874079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:38.874036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:39.144503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:39.144481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"]
Sep 05 02:19:39.146517 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:39.146494    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bceb5c2_cca1_49e0_91f0_9739f956bf2a.slice/crio-3a736cbdfb19d8d1a8ed6854b4c758a0b901ae8b194f1c87d88765ce205fd33a WatchSource:0}: Error finding container 3a736cbdfb19d8d1a8ed6854b4c758a0b901ae8b194f1c87d88765ce205fd33a: Status 404 returned error can't find the container with id 3a736cbdfb19d8d1a8ed6854b4c758a0b901ae8b194f1c87d88765ce205fd33a
Sep 05 02:19:39.519793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:39.519709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm" event={"ID":"8bceb5c2-cca1-49e0-91f0-9739f956bf2a","Type":"ContainerStarted","Data":"3a736cbdfb19d8d1a8ed6854b4c758a0b901ae8b194f1c87d88765ce205fd33a"}
Sep 05 02:19:40.525023 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.524939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm" event={"ID":"8bceb5c2-cca1-49e0-91f0-9739f956bf2a","Type":"ContainerStarted","Data":"d3482e9500ac8842c4ae954b3c5b02c52a0b11e4498bf654dea7c319a6e28bd5"}
Sep 05 02:19:40.525384 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.525092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:40.529513 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.529494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm"
Sep 05 02:19:40.542131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.542089    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8qvbm" podStartSLOduration=2.488202621 podStartE2EDuration="3.542076254s" totalImagesPullingTime="1.053873633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:40.541339492 +0000 UTC m=+124.180551315" watchObservedRunningTime="2026-09-05 02:19:40.542076254 +0000 UTC m=+124.181288082"
Sep 05 02:19:40.740322 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.740290    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:19:40.762331 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.762304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:19:40.762463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.762392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.765106 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765072    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:19:40.765247 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765126    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:19:40.765334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:19:40.765466 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765446    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:19:40.765519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765478    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-74hz2\""
Sep 05 02:19:40.765620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765602    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:19:40.765732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765716    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:19:40.765812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.765799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:19:40.932162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.932264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.932264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.932264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.932365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kql\" (UniqueName: \"kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:40.932365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:40.932324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033602 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kql\" (UniqueName: \"kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033718 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033718 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033855 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033920 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.033972 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.033939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.034335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.034310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.034427 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.034378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.034556 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.034535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.036171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.036147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.036252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.036238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.067810 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.067790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kql\" (UniqueName: \"kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql\") pod \"console-5ff6d47b8f-84hj9\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") " pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.071333 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.071308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:41.362826 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.362759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:19:41.365057 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:41.365029    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092dd973_6986_449b_88ae_665ee06426d8.slice/crio-f7c70eeba5026189df04db1e20cbf4fc3c67ea7ad17aab3224b602fe0af7a5c8 WatchSource:0}: Error finding container f7c70eeba5026189df04db1e20cbf4fc3c67ea7ad17aab3224b602fe0af7a5c8: Status 404 returned error can't find the container with id f7c70eeba5026189df04db1e20cbf4fc3c67ea7ad17aab3224b602fe0af7a5c8
Sep 05 02:19:41.529588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:41.529556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6d47b8f-84hj9" event={"ID":"092dd973-6986-449b-88ae-665ee06426d8","Type":"ContainerStarted","Data":"f7c70eeba5026189df04db1e20cbf4fc3c67ea7ad17aab3224b602fe0af7a5c8"}
Sep 05 02:19:44.541971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:44.541934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6d47b8f-84hj9" event={"ID":"092dd973-6986-449b-88ae-665ee06426d8","Type":"ContainerStarted","Data":"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"}
Sep 05 02:19:44.568450 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:44.568407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff6d47b8f-84hj9" podStartSLOduration=2.083352721 podStartE2EDuration="4.568395185s" totalImagesPullingTime="2.485042464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:44.568021158 +0000 UTC m=+128.207232977" watchObservedRunningTime="2026-09-05 02:19:44.568395185 +0000 UTC m=+128.207606985"
Sep 05 02:19:46.574543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:46.574508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:19:46.576705 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:46.576683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc83524-5eb0-4089-8aae-08ff8edd38c6-metrics-certs\") pod \"network-metrics-daemon-mj9x2\" (UID: \"fcc83524-5eb0-4089-8aae-08ff8edd38c6\") " pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:19:46.877814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:46.877731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptsx2\""
Sep 05 02:19:46.886253 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:46.886237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mj9x2"
Sep 05 02:19:47.156542 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:47.156435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mj9x2"]
Sep 05 02:19:47.159162 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:47.159135    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc83524_5eb0_4089_8aae_08ff8edd38c6.slice/crio-b5c8d72a3200cc331a05be83d9d5b32934498c215327ded8d926009916ceffbf WatchSource:0}: Error finding container b5c8d72a3200cc331a05be83d9d5b32934498c215327ded8d926009916ceffbf: Status 404 returned error can't find the container with id b5c8d72a3200cc331a05be83d9d5b32934498c215327ded8d926009916ceffbf
Sep 05 02:19:47.553527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:47.553447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mj9x2" event={"ID":"fcc83524-5eb0-4089-8aae-08ff8edd38c6","Type":"ContainerStarted","Data":"b5c8d72a3200cc331a05be83d9d5b32934498c215327ded8d926009916ceffbf"}
Sep 05 02:19:48.558152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:48.558119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mj9x2" event={"ID":"fcc83524-5eb0-4089-8aae-08ff8edd38c6","Type":"ContainerStarted","Data":"a658732bef700c129c062aa465289ed1299e5a2707648f013de30b6432802fab"}
Sep 05 02:19:48.558493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:48.558168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mj9x2" event={"ID":"fcc83524-5eb0-4089-8aae-08ff8edd38c6","Type":"ContainerStarted","Data":"8e2ac5af90bade29424e436aca29c0ba9ddbda8a00651838bd963bdc19a6a701"}
Sep 05 02:19:48.577266 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:48.577224    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mj9x2" podStartSLOduration=130.741290335 podStartE2EDuration="2m11.577210633s" totalImagesPullingTime="835.920298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:48.57543719 +0000 UTC m=+132.214648989" watchObservedRunningTime="2026-09-05 02:19:48.577210633 +0000 UTC m=+132.216422433"
Sep 05 02:19:49.320278 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.320244    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:19:49.323201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.323185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.330959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.330934    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 02:19:49.334420 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.334401    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:19:49.395525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lc5k\" (UniqueName: \"kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395755 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395755 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.395825 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.395732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.496948 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.496925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lc5k\" (UniqueName: \"kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497018 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.496956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497018 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.496971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497018 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.496991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497982 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.497982 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.497938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.499262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.499241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.499481 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.499459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.505474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.505456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lc5k\" (UniqueName: \"kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k\") pod \"console-bdc5b496d-lh4wf\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") " pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.632137 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.632070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:49.903390 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:49.903358    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:19:49.906822 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:49.906796    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d00f560_9b42_4103_8dc7_e6b293264225.slice/crio-a32e9c8c0fe7373dfa8847605d6b91dae65d623e4b960aa6c1350b4194c11b01 WatchSource:0}: Error finding container a32e9c8c0fe7373dfa8847605d6b91dae65d623e4b960aa6c1350b4194c11b01: Status 404 returned error can't find the container with id a32e9c8c0fe7373dfa8847605d6b91dae65d623e4b960aa6c1350b4194c11b01
Sep 05 02:19:50.566288 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:50.566251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc5b496d-lh4wf" event={"ID":"4d00f560-9b42-4103-8dc7-e6b293264225","Type":"ContainerStarted","Data":"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"}
Sep 05 02:19:50.566436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:50.566308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc5b496d-lh4wf" event={"ID":"4d00f560-9b42-4103-8dc7-e6b293264225","Type":"ContainerStarted","Data":"a32e9c8c0fe7373dfa8847605d6b91dae65d623e4b960aa6c1350b4194c11b01"}
Sep 05 02:19:50.585045 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:50.584997    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bdc5b496d-lh4wf" podStartSLOduration=1.584981626 podStartE2EDuration="1.584981626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:50.584069708 +0000 UTC m=+134.223281527" watchObservedRunningTime="2026-09-05 02:19:50.584981626 +0000 UTC m=+134.224193426"
Sep 05 02:19:51.072379 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:51.072346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:51.072842 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:51.072405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:51.076930 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:51.076907    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:51.572874 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:51.572845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:19:52.380569 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.380536    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"]
Sep 05 02:19:52.384876 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.384849    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.387656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.387633    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 05 02:19:52.389002 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.388867    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:52.389002 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.388876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tdv7x\""
Sep 05 02:19:52.389002 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.388908    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:52.396305 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.396284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"]
Sep 05 02:19:52.407712 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.407688    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cbmls"]
Sep 05 02:19:52.412162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.412142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.414843 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.414823    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:52.415162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.415138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:52.415435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.415300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:52.415435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.415319    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dlqp5\""
Sep 05 02:19:52.419064 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.419036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.419162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.419093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec34bf33-7aca-414b-8d88-5cdd084b047d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.419252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.419229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.419351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.419265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsh6p\" (UniqueName: \"kubernetes.io/projected/ec34bf33-7aca-414b-8d88-5cdd084b047d-kube-api-access-vsh6p\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.519699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsh6p\" (UniqueName: \"kubernetes.io/projected/ec34bf33-7aca-414b-8d88-5cdd084b047d-kube-api-access-vsh6p\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-textfile\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.519854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-wtmp\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:52.519900    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:52.519963    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls podName:ec34bf33-7aca-414b-8d88-5cdd084b047d nodeName:}" failed. No retries permitted until 2026-09-05 02:19:53.019943827 +0000 UTC m=+136.659155611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-vmmsf" (UID: "ec34bf33-7aca-414b-8d88-5cdd084b047d") : secret "openshift-state-metrics-tls" not found
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.519984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec34bf33-7aca-414b-8d88-5cdd084b047d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjhl\" (UniqueName: \"kubernetes.io/projected/2da229ad-b5f6-4656-82a3-57033b84b982-kube-api-access-jfjhl\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-sys\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-metrics-client-ca\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-root\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.520692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.520670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec34bf33-7aca-414b-8d88-5cdd084b047d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.522198 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.522172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.530527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.530499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsh6p\" (UniqueName: \"kubernetes.io/projected/ec34bf33-7aca-414b-8d88-5cdd084b047d-kube-api-access-vsh6p\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:52.621351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjhl\" (UniqueName: \"kubernetes.io/projected/2da229ad-b5f6-4656-82a3-57033b84b982-kube-api-access-jfjhl\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-sys\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-metrics-client-ca\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-sys\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-root\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-root\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.621846 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:52.621681    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:19:52.621846 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:52.621764    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls podName:2da229ad-b5f6-4656-82a3-57033b84b982 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:53.121730726 +0000 UTC m=+136.760942511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls") pod "node-exporter-cbmls" (UID: "2da229ad-b5f6-4656-82a3-57033b84b982") : secret "node-exporter-tls" not found
Sep 05 02:19:52.621976 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.621591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622037 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-textfile\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622094 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-wtmp\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622142 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-metrics-client-ca\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622353    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-textfile\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.622480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.622401    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-wtmp\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.624120 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.624100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:52.632793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:52.632723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjhl\" (UniqueName: \"kubernetes.io/projected/2da229ad-b5f6-4656-82a3-57033b84b982-kube-api-access-jfjhl\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:53.025533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.025501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:53.027923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.027897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec34bf33-7aca-414b-8d88-5cdd084b047d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vmmsf\" (UID: \"ec34bf33-7aca-414b-8d88-5cdd084b047d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:53.126592 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.126547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:53.128831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.128807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2da229ad-b5f6-4656-82a3-57033b84b982-node-exporter-tls\") pod \"node-exporter-cbmls\" (UID: \"2da229ad-b5f6-4656-82a3-57033b84b982\") " pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:53.296523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.296451    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"
Sep 05 02:19:53.321956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.321912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmls"
Sep 05 02:19:53.334808 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:53.334764    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da229ad_b5f6_4656_82a3_57033b84b982.slice/crio-5ffae4b9ed1abfaf66e86eb9d50b55b1a7d873d566c03f0448841210a2c311c9 WatchSource:0}: Error finding container 5ffae4b9ed1abfaf66e86eb9d50b55b1a7d873d566c03f0448841210a2c311c9: Status 404 returned error can't find the container with id 5ffae4b9ed1abfaf66e86eb9d50b55b1a7d873d566c03f0448841210a2c311c9
Sep 05 02:19:53.579878 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.579806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmls" event={"ID":"2da229ad-b5f6-4656-82a3-57033b84b982","Type":"ContainerStarted","Data":"5ffae4b9ed1abfaf66e86eb9d50b55b1a7d873d566c03f0448841210a2c311c9"}
Sep 05 02:19:53.580970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:53.580937    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf"]
Sep 05 02:19:53.583708 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:53.583681    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec34bf33_7aca_414b_8d88_5cdd084b047d.slice/crio-25912b88ae331e4ab6434fab909f32222b270fc86357ad547b1551ac490d7791 WatchSource:0}: Error finding container 25912b88ae331e4ab6434fab909f32222b270fc86357ad547b1551ac490d7791: Status 404 returned error can't find the container with id 25912b88ae331e4ab6434fab909f32222b270fc86357ad547b1551ac490d7791
Sep 05 02:19:54.083905 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.083872    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bd46df57-gpl5k"]
Sep 05 02:19:54.084201 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:54.084176    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-67bd46df57-gpl5k" podUID="a57eb46c-4ae4-4b55-9d65-57c801029bae"
Sep 05 02:19:54.585834 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.585797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf" event={"ID":"ec34bf33-7aca-414b-8d88-5cdd084b047d","Type":"ContainerStarted","Data":"ebd44b82ba85202b327fd970fd9f0e057863bc0e0228e097d737bc0c662978bc"}
Sep 05 02:19:54.586191 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.585874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf" event={"ID":"ec34bf33-7aca-414b-8d88-5cdd084b047d","Type":"ContainerStarted","Data":"2b79b3dc9356648a6ebe9a6492e617a5b90585505a6b5743c61516b2d2f28aa0"}
Sep 05 02:19:54.586191 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.585906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf" event={"ID":"ec34bf33-7aca-414b-8d88-5cdd084b047d","Type":"ContainerStarted","Data":"25912b88ae331e4ab6434fab909f32222b270fc86357ad547b1551ac490d7791"}
Sep 05 02:19:54.587496 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.587471    2560 generic.go:356] "Generic (PLEG): container finished" podID="2da229ad-b5f6-4656-82a3-57033b84b982" containerID="c7f648d6c6d5944f79c26cf9c70efa7a133eae410e2b5750ffcf7e71f0db696a" exitCode=0
Sep 05 02:19:54.587643 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.587561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmls" event={"ID":"2da229ad-b5f6-4656-82a3-57033b84b982","Type":"ContainerDied","Data":"c7f648d6c6d5944f79c26cf9c70efa7a133eae410e2b5750ffcf7e71f0db696a"}
Sep 05 02:19:54.587643 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.587633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:19:54.594129 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.594111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:19:54.641396 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641423    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641450    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641481    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hqf7\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641704 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641774 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641842 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:19:54.641842 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets\") pod \"a57eb46c-4ae4-4b55-9d65-57c801029bae\" (UID: \"a57eb46c-4ae4-4b55-9d65-57c801029bae\") "
Sep 05 02:19:54.641944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:54.642007 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.641981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:54.642927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.642735    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a57eb46c-4ae4-4b55-9d65-57c801029bae-ca-trust-extracted\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.642927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.642780    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-trusted-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.642927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.642795    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-certificates\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.644018 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.643989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:54.644120 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.644019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7" (OuterVolumeSpecName: "kube-api-access-7hqf7") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "kube-api-access-7hqf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:54.644198 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.644149    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:54.644427 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.644403    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a57eb46c-4ae4-4b55-9d65-57c801029bae" (UID: "a57eb46c-4ae4-4b55-9d65-57c801029bae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:54.743923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.743894    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqf7\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-kube-api-access-7hqf7\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.743923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.743923    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-bound-sa-token\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.744058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.743938    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-installation-pull-secrets\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:54.744058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:54.743953    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a57eb46c-4ae4-4b55-9d65-57c801029bae-image-registry-private-configuration\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:55.593402 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.593357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf" event={"ID":"ec34bf33-7aca-414b-8d88-5cdd084b047d","Type":"ContainerStarted","Data":"40cdeaa4278d3bcf0e64c2fe5704565746acf017170908f972c4bea25e1f8bf6"}
Sep 05 02:19:55.595948 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.595928    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd46df57-gpl5k"
Sep 05 02:19:55.596037 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.595929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmls" event={"ID":"2da229ad-b5f6-4656-82a3-57033b84b982","Type":"ContainerStarted","Data":"837eb1c4a42be5a17da7c4aa7cb643da25ec9ed4ae8d3b440beeed379e56f4aa"}
Sep 05 02:19:55.596081 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.596049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmls" event={"ID":"2da229ad-b5f6-4656-82a3-57033b84b982","Type":"ContainerStarted","Data":"d875637e5e1a778bd6f71891b6199a61a909529b228dff293772a97e5ecc14f5"}
Sep 05 02:19:55.616004 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.615968    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vmmsf" podStartSLOduration=1.847143881 podStartE2EDuration="3.615955167s" totalImagesPullingTime="1.768811286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:55.615325454 +0000 UTC m=+139.254537275" watchObservedRunningTime="2026-09-05 02:19:55.615955167 +0000 UTC m=+139.255166967"
Sep 05 02:19:55.645210 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.645185    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bd46df57-gpl5k"]
Sep 05 02:19:55.652729 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.652708    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67bd46df57-gpl5k"]
Sep 05 02:19:55.675715 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.675536    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cbmls" podStartSLOduration=2.710939109 podStartE2EDuration="3.67552352s" totalImagesPullingTime="902.746326ms" totalInitContainerRuntime="61.838085ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:55.675159278 +0000 UTC m=+139.314371106" watchObservedRunningTime="2026-09-05 02:19:55.67552352 +0000 UTC m=+139.314735384"
Sep 05 02:19:55.752166 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:55.752117    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a57eb46c-4ae4-4b55-9d65-57c801029bae-registry-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.484226 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.484193    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-544c549dfd-rsz54"]
Sep 05 02:19:56.487492 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.487475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.490215 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.490191    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 05 02:19:56.490925 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.490909    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1316s0cnvsdgt\""
Sep 05 02:19:56.491636 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.491612    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 05 02:19:56.491772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.491642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 05 02:19:56.491772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.491644    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zqtfs\""
Sep 05 02:19:56.491772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.491717    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 05 02:19:56.491772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.491611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 05 02:19:56.505436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.505415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-544c549dfd-rsz54"]
Sep 05 02:19:56.556751 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556835 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s489d\" (UniqueName: \"kubernetes.io/projected/df6028fe-8e6d-4c74-9d01-b66f396b668c-kube-api-access-s489d\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556890 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556890 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df6028fe-8e6d-4c74-9d01-b66f396b668c-metrics-client-ca\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.556970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.557075 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.556997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-grpc-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657596 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df6028fe-8e6d-4c74-9d01-b66f396b668c-metrics-client-ca\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-grpc-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.657957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.657854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s489d\" (UniqueName: \"kubernetes.io/projected/df6028fe-8e6d-4c74-9d01-b66f396b668c-kube-api-access-s489d\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.658464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.658437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df6028fe-8e6d-4c74-9d01-b66f396b668c-metrics-client-ca\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.660475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.660449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.660577 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.660485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-grpc-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.660652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.660574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.660863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.660837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.661055 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.661036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.661211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.661188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df6028fe-8e6d-4c74-9d01-b66f396b668c-secret-thanos-querier-tls\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.666650 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.666629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s489d\" (UniqueName: \"kubernetes.io/projected/df6028fe-8e6d-4c74-9d01-b66f396b668c-kube-api-access-s489d\") pod \"thanos-querier-544c549dfd-rsz54\" (UID: \"df6028fe-8e6d-4c74-9d01-b66f396b668c\") " pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.744156 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.744091    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"]
Sep 05 02:19:56.748239 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.748224    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.752684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.752664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 05 02:19:56.753009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.752991    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vg22m\""
Sep 05 02:19:56.753088 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.753012    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 05 02:19:56.753263 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.753246    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 05 02:19:56.753516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.753501    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b1ntr7e4qfms4\""
Sep 05 02:19:56.753886 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.753868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 05 02:19:56.770904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.770884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"]
Sep 05 02:19:56.796910 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.796888    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:19:56.859203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-audit-log\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859300 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-tls\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-metrics-server-audit-profiles\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859412 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-client-certs\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859443 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-client-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.859476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.859452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25td7\" (UniqueName: \"kubernetes.io/projected/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-kube-api-access-25td7\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960762 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-tls\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-metrics-server-audit-profiles\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-client-certs\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-client-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25td7\" (UniqueName: \"kubernetes.io/projected/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-kube-api-access-25td7\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.960907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-audit-log\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.961152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.960898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.961357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.961322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-audit-log\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.961709 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.961682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.961836 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.961816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-metrics-server-audit-profiles\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.963267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.963238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-tls\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.963387 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.963313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-client-ca-bundle\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.963446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.963402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-secret-metrics-server-client-certs\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.969132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.969111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25td7\" (UniqueName: \"kubernetes.io/projected/7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8-kube-api-access-25td7\") pod \"metrics-server-5b7fb5cbb7-7hrtt\" (UID: \"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8\") " pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:56.971313 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:56.971289    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a57eb46c-4ae4-4b55-9d65-57c801029bae" path="/var/lib/kubelet/pods/a57eb46c-4ae4-4b55-9d65-57c801029bae/volumes"
Sep 05 02:19:57.057781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.057728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:19:57.077392 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.074469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-544c549dfd-rsz54"]
Sep 05 02:19:57.183699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.183669    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"]
Sep 05 02:19:57.188021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.188006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.190550 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.190530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 05 02:19:57.190655 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.190551    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rgzdn\""
Sep 05 02:19:57.202846 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.202822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"]
Sep 05 02:19:57.263366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.263332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.263505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.263389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjz2l\" (UniqueName: \"kubernetes.io/projected/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-kube-api-access-vjz2l\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.331225 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.331165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"]
Sep 05 02:19:57.334316 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:57.334290    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0d9bfa_071e_4eca_94cc_5e1fea04a5e8.slice/crio-c5f588faa19159e272999014b463a80bbe17168edcd1052d9c3021fbd2ddc819 WatchSource:0}: Error finding container c5f588faa19159e272999014b463a80bbe17168edcd1052d9c3021fbd2ddc819: Status 404 returned error can't find the container with id c5f588faa19159e272999014b463a80bbe17168edcd1052d9c3021fbd2ddc819
Sep 05 02:19:57.364279 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.364254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.364369 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.364293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz2l\" (UniqueName: \"kubernetes.io/projected/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-kube-api-access-vjz2l\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.364438 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:57.364420    2560 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 05 02:19:57.364493 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:19:57.364483    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert podName:6d5c85dc-3cea-4c2f-9533-8592ecbfe73f nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.864462668 +0000 UTC m=+141.503674453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-c6s24" (UID: "6d5c85dc-3cea-4c2f-9533-8592ecbfe73f") : secret "monitoring-plugin-cert" not found
Sep 05 02:19:57.373567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.373544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz2l\" (UniqueName: \"kubernetes.io/projected/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-kube-api-access-vjz2l\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.604314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.604224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt" event={"ID":"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8","Type":"ContainerStarted","Data":"c5f588faa19159e272999014b463a80bbe17168edcd1052d9c3021fbd2ddc819"}
Sep 05 02:19:57.605672 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.605646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"4a50e1daceb43fb9dc34676747cf1da825ceedd64c240dd2ac90a577a439a053"}
Sep 05 02:19:57.867348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.867258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.870180 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.870146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d5c85dc-3cea-4c2f-9533-8592ecbfe73f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-c6s24\" (UID: \"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:57.931501 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.931473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:19:57.971975 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.971943    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:19:57.975689 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.975656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:57.984720 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:57.984638    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:19:58.068703 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.068883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.068883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.068883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8k2x\" (UniqueName: \"kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.068883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.069102 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.069102 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.068946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.098605 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.098573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:19:58.170351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8k2x\" (UniqueName: \"kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170519 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.170765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.170598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.171172 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.171127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.171404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.171380    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.171529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.171505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.171590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.171558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.173481 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.173456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.174452 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.174430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.179224 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.179199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8k2x\" (UniqueName: \"kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x\") pod \"console-765f6c74db-x565l\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") " pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.288261 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.288218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:19:58.734176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.734142    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:19:58.738610 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.738584    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.742686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.742500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 05 02:19:58.742686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.742534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 05 02:19:58.742686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.742578    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 05 02:19:58.742686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.742660    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 05 02:19:58.743052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.742756    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 05 02:19:58.743052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.743041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9p5c023envmqm\""
Sep 05 02:19:58.743936 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.743878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 05 02:19:58.744170 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.744114    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rcx7c\""
Sep 05 02:19:58.749695 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.749671    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 05 02:19:58.749826 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.749676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 05 02:19:58.750043 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.750021    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 05 02:19:58.750136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.750058    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 05 02:19:58.760299 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.760278    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 05 02:19:58.760525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.760506    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 05 02:19:58.760784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.760766    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 05 02:19:58.763491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.763473    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 05 02:19:58.775647 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.775924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776123 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776123 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.775976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776123 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5crt\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.776883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.776865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:19:58.877660 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.877660 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.877983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5crt\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.878176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.879297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.879297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.879297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.879297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.878735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.879496 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.879366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.880099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.880063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.880875 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.880852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.881633 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.881482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.882781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.882038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.882781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.882059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.882781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.882443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.882975 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.882891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.884534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.883290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.884534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.883758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.884534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.884260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.884534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.884364    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.884534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.884459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.885212 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.885188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.885610 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.885588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.886275 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.886252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:58.890646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:58.890629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5crt\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt\") pod \"prometheus-k8s-0\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:59.050106 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.050082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:19:59.302052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.302022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"]
Sep 05 02:19:59.304249 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:59.304192    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5c85dc_3cea_4c2f_9533_8592ecbfe73f.slice/crio-53a1814ce8d716d254e20c086f4358d3d1d4f7f45dd6152044c5f3a8b9dc83e4 WatchSource:0}: Error finding container 53a1814ce8d716d254e20c086f4358d3d1d4f7f45dd6152044c5f3a8b9dc83e4: Status 404 returned error can't find the container with id 53a1814ce8d716d254e20c086f4358d3d1d4f7f45dd6152044c5f3a8b9dc83e4
Sep 05 02:19:59.304662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.304642    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:19:59.306172 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:59.306155    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f846dc_5443_4012_abf5_06b1b5b248df.slice/crio-e090865f4c4fbd2aff1d1b7b2a8b169d0b4aca20db2e56710da3774d3ffe2e6a WatchSource:0}: Error finding container e090865f4c4fbd2aff1d1b7b2a8b169d0b4aca20db2e56710da3774d3ffe2e6a: Status 404 returned error can't find the container with id e090865f4c4fbd2aff1d1b7b2a8b169d0b4aca20db2e56710da3774d3ffe2e6a
Sep 05 02:19:59.351502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.351456    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:19:59.354632 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:19:59.354609    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode215d34f_4bfd_4d9d_9d69_18860a39e8c9.slice/crio-8953ecf3d60fdc015cb47a01c4ecdc07128393a8a00783e0a446f1a9f1743f11 WatchSource:0}: Error finding container 8953ecf3d60fdc015cb47a01c4ecdc07128393a8a00783e0a446f1a9f1743f11: Status 404 returned error can't find the container with id 8953ecf3d60fdc015cb47a01c4ecdc07128393a8a00783e0a446f1a9f1743f11
Sep 05 02:19:59.615440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.615359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"bd3153d091b125e6a21dd24c5aa5eb72d7e5317010a982c96ece439d2cd69aab"}
Sep 05 02:19:59.615440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.615422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"ae7d8d5e0d85bab6403ddbc8f4a0037efefeab596dca8a7297c6a807f7849e0c"}
Sep 05 02:19:59.615625 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.615458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"26250db6ed8b404115432e7d8a21d70138e935fd2a4692f1b5113ae40718ed6f"}
Sep 05 02:19:59.616485 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.616461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24" event={"ID":"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f","Type":"ContainerStarted","Data":"53a1814ce8d716d254e20c086f4358d3d1d4f7f45dd6152044c5f3a8b9dc83e4"}
Sep 05 02:19:59.617490 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.617467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"8953ecf3d60fdc015cb47a01c4ecdc07128393a8a00783e0a446f1a9f1743f11"}
Sep 05 02:19:59.618897 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.618876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f6c74db-x565l" event={"ID":"94f846dc-5443-4012-abf5-06b1b5b248df","Type":"ContainerStarted","Data":"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"}
Sep 05 02:19:59.618967 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.618912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f6c74db-x565l" event={"ID":"94f846dc-5443-4012-abf5-06b1b5b248df","Type":"ContainerStarted","Data":"e090865f4c4fbd2aff1d1b7b2a8b169d0b4aca20db2e56710da3774d3ffe2e6a"}
Sep 05 02:19:59.620278 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.620260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt" event={"ID":"7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8","Type":"ContainerStarted","Data":"f721cd9d55070e33a449f5a43f1a463bab8e7b04a6145a979f4c67f0f93c907a"}
Sep 05 02:19:59.632978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.632962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:19:59.639086 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.639051    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765f6c74db-x565l" podStartSLOduration=2.639040347 podStartE2EDuration="2.639040347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:59.638605877 +0000 UTC m=+143.277817698" watchObservedRunningTime="2026-09-05 02:19:59.639040347 +0000 UTC m=+143.278252146"
Sep 05 02:19:59.662334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:19:59.662296    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt" podStartSLOduration=1.968660951 podStartE2EDuration="3.662283218s" totalImagesPullingTime="1.693622267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:59.66190335 +0000 UTC m=+143.301115176" watchObservedRunningTime="2026-09-05 02:19:59.662283218 +0000 UTC m=+143.301495019"
Sep 05 02:20:01.631859 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.631821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"b77539a2dc119e8b6fd14a4aa8910bc1b65f4ad1b54b4f853a193502a746d9b3"}
Sep 05 02:20:01.632314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.631890    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"a10325a23f13cf835d54bdb7a67218308aae7e5a299e888d0ecaf7e4f847cbb5"}
Sep 05 02:20:01.632314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.631923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" event={"ID":"df6028fe-8e6d-4c74-9d01-b66f396b668c","Type":"ContainerStarted","Data":"e0c5e988e9c012d4f2f7e4d6af38ce2c98ebc2e1da541c7fa43a0c57cd9111fc"}
Sep 05 02:20:01.632314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.631970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:20:01.633381 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.633357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24" event={"ID":"6d5c85dc-3cea-4c2f-9533-8592ecbfe73f","Type":"ContainerStarted","Data":"07a342fa98b024117d09dd90ace6bbd7adedacd8713c6f3079356dfdaaeabca0"}
Sep 05 02:20:01.633485 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.633401    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:20:01.635068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.635039    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e" exitCode=0
Sep 05 02:20:01.635161 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.635093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"}
Sep 05 02:20:01.638756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.638728    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24"
Sep 05 02:20:01.657372 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.657329    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54" podStartSLOduration=2.243283925 podStartE2EDuration="5.657315605s" totalImagesPullingTime="3.41403168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:01.656292983 +0000 UTC m=+145.295504816" watchObservedRunningTime="2026-09-05 02:20:01.657315605 +0000 UTC m=+145.296527405"
Sep 05 02:20:01.707446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:01.707404    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-c6s24" podStartSLOduration=3.287414875 podStartE2EDuration="4.707391148s" totalImagesPullingTime="1.419976273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:01.705856738 +0000 UTC m=+145.345068570" watchObservedRunningTime="2026-09-05 02:20:01.707391148 +0000 UTC m=+145.346602948"
Sep 05 02:20:05.659142 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"}
Sep 05 02:20:05.659565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"}
Sep 05 02:20:05.659565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"}
Sep 05 02:20:05.659565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"}
Sep 05 02:20:05.659565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"}
Sep 05 02:20:05.659565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.659367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerStarted","Data":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"}
Sep 05 02:20:05.712367 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:05.712316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.3024973 podStartE2EDuration="7.712300212s" totalImagesPullingTime="4.384130289s" totalInitContainerRuntime="25.672623ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:05.710622103 +0000 UTC m=+149.349833922" watchObservedRunningTime="2026-09-05 02:20:05.712300212 +0000 UTC m=+149.351512012"
Sep 05 02:20:07.646146 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:07.646113    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-544c549dfd-rsz54"
Sep 05 02:20:08.288858 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:08.288828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:20:08.289007 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:08.288881    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:20:08.293477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:08.293456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:20:08.674189 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:08.674163    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:20:08.764498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:08.764471    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:20:09.050703 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:09.050626    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:17.058915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:17.058885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:20:17.059291 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:17.058937    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:20:18.716165 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:18.716132    2560 generic.go:356] "Generic (PLEG): container finished" podID="77ab1399-a71b-48c6-83d3-0b1d33539384" containerID="ca09d9dcf121d7c82994165d0f4077a5151b89e0e378d549b055ce4e93115fe8" exitCode=0
Sep 05 02:20:18.716546 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:18.716171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw" event={"ID":"77ab1399-a71b-48c6-83d3-0b1d33539384","Type":"ContainerDied","Data":"ca09d9dcf121d7c82994165d0f4077a5151b89e0e378d549b055ce4e93115fe8"}
Sep 05 02:20:18.716546 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:18.716471    2560 scope.go:117] "RemoveContainer" containerID="ca09d9dcf121d7c82994165d0f4077a5151b89e0e378d549b055ce4e93115fe8"
Sep 05 02:20:19.722550 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:19.722511    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vltzw" event={"ID":"77ab1399-a71b-48c6-83d3-0b1d33539384","Type":"ContainerStarted","Data":"f0fc3a24c8fdbed60b29d62bc628d9e852ce96fcfa685ed83b2000853e6c78ca"}
Sep 05 02:20:22.958483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:22.958441    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bdc5b496d-lh4wf" podUID="4d00f560-9b42-4103-8dc7-e6b293264225" containerName="console" containerID="cri-o://ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3" gracePeriod=15
Sep 05 02:20:23.199750 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.199720    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:20:23.280933 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.280905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.280946    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.280991    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lc5k\" (UniqueName: \"kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert\") pod \"4d00f560-9b42-4103-8dc7-e6b293264225\" (UID: \"4d00f560-9b42-4103-8dc7-e6b293264225\") "
Sep 05 02:20:23.281514 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281478    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:23.281514 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config" (OuterVolumeSpecName: "console-config") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:23.281795 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281514    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:23.281795 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.281765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:23.283252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.283231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:23.283544 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.283520    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:23.283603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.283529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k" (OuterVolumeSpecName: "kube-api-access-8lc5k") pod "4d00f560-9b42-4103-8dc7-e6b293264225" (UID: "4d00f560-9b42-4103-8dc7-e6b293264225"). InnerVolumeSpecName "kube-api-access-8lc5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:23.382209 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382184    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-service-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382209 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382206    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-console-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382215    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382225    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-oauth-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382233    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d00f560-9b42-4103-8dc7-e6b293264225-console-oauth-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382242    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d00f560-9b42-4103-8dc7-e6b293264225-trusted-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.382356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.382250    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lc5k\" (UniqueName: \"kubernetes.io/projected/4d00f560-9b42-4103-8dc7-e6b293264225-kube-api-access-8lc5k\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:23.741885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.741856    2560 generic.go:356] "Generic (PLEG): container finished" podID="4d00f560-9b42-4103-8dc7-e6b293264225" containerID="ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3" exitCode=0
Sep 05 02:20:23.742053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.741913    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc5b496d-lh4wf"
Sep 05 02:20:23.742053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.741949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc5b496d-lh4wf" event={"ID":"4d00f560-9b42-4103-8dc7-e6b293264225","Type":"ContainerDied","Data":"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"}
Sep 05 02:20:23.742053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.742008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc5b496d-lh4wf" event={"ID":"4d00f560-9b42-4103-8dc7-e6b293264225","Type":"ContainerDied","Data":"a32e9c8c0fe7373dfa8847605d6b91dae65d623e4b960aa6c1350b4194c11b01"}
Sep 05 02:20:23.742053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.742037    2560 scope.go:117] "RemoveContainer" containerID="ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"
Sep 05 02:20:23.749569 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.749541    2560 generic.go:356] "Generic (PLEG): container finished" podID="21a849dc-df64-4332-803b-45cdb526a487" containerID="1b4b75536faa826e3b29313407edff7568b22e1f420d6880357281137da60aaa" exitCode=0
Sep 05 02:20:23.749700 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.749645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp" event={"ID":"21a849dc-df64-4332-803b-45cdb526a487","Type":"ContainerDied","Data":"1b4b75536faa826e3b29313407edff7568b22e1f420d6880357281137da60aaa"}
Sep 05 02:20:23.750072 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.750048    2560 scope.go:117] "RemoveContainer" containerID="1b4b75536faa826e3b29313407edff7568b22e1f420d6880357281137da60aaa"
Sep 05 02:20:23.759786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.759767    2560 scope.go:117] "RemoveContainer" containerID="ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"
Sep 05 02:20:23.760064 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:20:23.760044    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3\": container with ID starting with ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3 not found: ID does not exist" containerID="ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"
Sep 05 02:20:23.760124 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.760071    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3"} err="failed to get container status \"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3\": rpc error: code = NotFound desc = could not find container \"ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3\": container with ID starting with ce889506142a7fe206ea9b9f0c99e976dd51f4a9973ac28212bf73d740126ba3 not found: ID does not exist"
Sep 05 02:20:23.785246 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.785221    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:20:23.796718 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:23.796653    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bdc5b496d-lh4wf"]
Sep 05 02:20:24.756962 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:24.756927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-hp7jp" event={"ID":"21a849dc-df64-4332-803b-45cdb526a487","Type":"ContainerStarted","Data":"f06e8442d323839edd99ea767de5617a96c8fb9c0dd861362942d145526af136"}
Sep 05 02:20:24.970095 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:24.970062    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d00f560-9b42-4103-8dc7-e6b293264225" path="/var/lib/kubelet/pods/4d00f560-9b42-4103-8dc7-e6b293264225/volumes"
Sep 05 02:20:33.783591 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:33.783507    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ff6d47b8f-84hj9" podUID="092dd973-6986-449b-88ae-665ee06426d8" containerName="console" containerID="cri-o://4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84" gracePeriod=15
Sep 05 02:20:34.078410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.078355    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:20:34.267143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6kql\" (UniqueName: \"kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267207    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267296    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config\") pod \"092dd973-6986-449b-88ae-665ee06426d8\" (UID: \"092dd973-6986-449b-88ae-665ee06426d8\") "
Sep 05 02:20:34.267676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267648    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:34.267676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:34.267830 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.267656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config" (OuterVolumeSpecName: "console-config") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:34.269243 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.269217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:34.269324 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.269235    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql" (OuterVolumeSpecName: "kube-api-access-f6kql") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "kube-api-access-f6kql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:34.269324 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.269239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "092dd973-6986-449b-88ae-665ee06426d8" (UID: "092dd973-6986-449b-88ae-665ee06426d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:34.368136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368063    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-service-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.368136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368101    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-oauth-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.368136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368114    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/092dd973-6986-449b-88ae-665ee06426d8-console-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.368136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368127    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-oauth-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.368136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368140    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6kql\" (UniqueName: \"kubernetes.io/projected/092dd973-6986-449b-88ae-665ee06426d8-kube-api-access-f6kql\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.368448 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.368156    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/092dd973-6986-449b-88ae-665ee06426d8-console-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:20:34.799725 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.799691    2560 generic.go:356] "Generic (PLEG): container finished" podID="092dd973-6986-449b-88ae-665ee06426d8" containerID="4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84" exitCode=0
Sep 05 02:20:34.800159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.799765    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6d47b8f-84hj9"
Sep 05 02:20:34.800159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.799780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6d47b8f-84hj9" event={"ID":"092dd973-6986-449b-88ae-665ee06426d8","Type":"ContainerDied","Data":"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"}
Sep 05 02:20:34.800159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.799834    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6d47b8f-84hj9" event={"ID":"092dd973-6986-449b-88ae-665ee06426d8","Type":"ContainerDied","Data":"f7c70eeba5026189df04db1e20cbf4fc3c67ea7ad17aab3224b602fe0af7a5c8"}
Sep 05 02:20:34.800159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.799854    2560 scope.go:117] "RemoveContainer" containerID="4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"
Sep 05 02:20:34.809552 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.809535    2560 scope.go:117] "RemoveContainer" containerID="4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"
Sep 05 02:20:34.809907 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:20:34.809861    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84\": container with ID starting with 4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84 not found: ID does not exist" containerID="4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"
Sep 05 02:20:34.809978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.809896    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84"} err="failed to get container status \"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84\": rpc error: code = NotFound desc = could not find container \"4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84\": container with ID starting with 4612fe518838a5290ae0f3b83ab052515a5bafb05ffe21d3ee3cb0ca97c9ec84 not found: ID does not exist"
Sep 05 02:20:34.824924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.824895    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:20:34.829691 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.829671    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff6d47b8f-84hj9"]
Sep 05 02:20:34.971031 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:34.970997    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092dd973-6986-449b-88ae-665ee06426d8" path="/var/lib/kubelet/pods/092dd973-6986-449b-88ae-665ee06426d8/volumes"
Sep 05 02:20:37.069246 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:37.069158    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:20:37.075162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:37.075139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b7fb5cbb7-7hrtt"
Sep 05 02:20:59.051227 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:59.051195    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:59.069488 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:59.069460    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:59.927030 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:20:59.927000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:17.159390 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159353    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159620    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="prometheus" containerID="cri-o://1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0" gracePeriod=600
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159650    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy" containerID="cri-o://f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952" gracePeriod=600
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159683    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="config-reloader" containerID="cri-o://5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1" gracePeriod=600
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159709    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-web" containerID="cri-o://c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f" gracePeriod=600
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159784    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e" gracePeriod=600
Sep 05 02:21:17.159880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.159666    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="thanos-sidecar" containerID="cri-o://68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2" gracePeriod=600
Sep 05 02:21:17.492255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.492228    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:17.586997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.586967    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.586997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587212    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587276    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.587568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587407    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587513    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587615    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587658    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5crt\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587730    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.587851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587841    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.588284 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.587865    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2\") pod \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\" (UID: \"e215d34f-4bfd-4d9d-9d69-18860a39e8c9\") "
Sep 05 02:21:17.588284 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.588130    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.588284 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.588147    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.588435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.588416    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.589185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.588927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:21:17.590079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.589718    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.590079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.589793    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.590079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.589839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:17.590079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.589839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.590366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.590366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.590652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590628    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.590923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590897    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config" (OuterVolumeSpecName: "config") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.591009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590986    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.591009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.590997    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.591347 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.591304    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:17.591466 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.591406    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out" (OuterVolumeSpecName: "config-out") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:21:17.591728 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.591703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.591935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.591912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.592824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.592802    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt" (OuterVolumeSpecName: "kube-api-access-h5crt") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "kube-api-access-h5crt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:17.602413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.602393    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config" (OuterVolumeSpecName: "web-config") pod "e215d34f-4bfd-4d9d-9d69-18860a39e8c9" (UID: "e215d34f-4bfd-4d9d-9d69-18860a39e8c9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688717    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-grpc-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688755    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688766    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-web-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688776    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688785    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.688793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688794    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5crt\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-kube-api-access-h5crt\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688802    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-config-out\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688810    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688818    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-kube-rbac-proxy\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688827    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688837    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-configmap-metrics-client-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688845    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688853    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688861    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-tls-assets\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688869    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688877    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688884    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-prometheus-k8s-db\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.689040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.688892    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e215d34f-4bfd-4d9d-9d69-18860a39e8c9-secret-metrics-client-certs\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983880    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983914    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983923    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983932    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983939    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983947    2560 generic.go:356] "Generic (PLEG): container finished" podID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0" exitCode=0
Sep 05 02:21:17.983968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983970    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.983972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984114    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"}
Sep 05 02:21:17.984281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.984182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e215d34f-4bfd-4d9d-9d69-18860a39e8c9","Type":"ContainerDied","Data":"8953ecf3d60fdc015cb47a01c4ecdc07128393a8a00783e0a446f1a9f1743f11"}
Sep 05 02:21:17.992892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.992877    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:17.999513 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:17.999497    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.006098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.006080    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.010504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.010483    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:18.013152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.013131    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.016433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.016413    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:18.020543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.020527    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.027680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.027656    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.034058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034042    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.034324 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.034305    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.034378 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034331    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.034378 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034348    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.034603 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.034587    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.034634 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034610    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.034634 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034625    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.034939 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.034917    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.035020 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034943    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.035020 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.034958    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.035184 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.035169    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.035282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035185    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.035282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035197    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.035449 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.035431    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.035531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035456    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.035531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035476    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.035860 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.035840    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.035919 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035863    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.035919 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.035877    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.036160 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:18.036143    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.036213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036164    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.036213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036179    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.036435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036416    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.036478 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036435    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.036676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036659    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.036719 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036677    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.036964 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036945    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.037043 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.036966    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.037209 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037192    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.037278 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037210    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.037444 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037427    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.037504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037445    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.037676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037653    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.037676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037673    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.037912 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037897    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.037912 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.037912    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.038177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038156    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.038177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038177    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.038451 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038432    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.038503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038452    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.038709 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038691    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.038765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038711    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.038954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038937    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.038996 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.038954    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.039194 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039177    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.039233 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039195    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.039424 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039402    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.039507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039423    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.039646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039626    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.039691 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039647    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.039891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039872    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.039963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.039891    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.040161 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040143    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.040212 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040161    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.040386 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040367    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.040432 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040387    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.040624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040606    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.040692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040625    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.040885 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040866    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.040936 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.040888    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.041122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041104    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.041167 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041123    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.041330 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041310    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.041375 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041331    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.041533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041518    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.041576 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041533    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.041794 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041776    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.041835 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041794    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.042009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.041993    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.042059 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.042008    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.042199 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.042182    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.042237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.042200    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.045935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.045911    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:18.046422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046393    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="prometheus"
Sep 05 02:21:18.046422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046413    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="prometheus"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046428    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="config-reloader"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046444    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="config-reloader"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046458    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-web"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046468    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-web"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046481    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="thanos-sidecar"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046496    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="thanos-sidecar"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046509    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046517    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046537    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="init-config-reloader"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046546    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="init-config-reloader"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046568    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092dd973-6986-449b-88ae-665ee06426d8" containerName="console"
Sep 05 02:21:18.046574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046576    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="092dd973-6986-449b-88ae-665ee06426d8" containerName="console"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046588    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-thanos"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046597    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-thanos"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046616    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d00f560-9b42-4103-8dc7-e6b293264225" containerName="console"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046624    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d00f560-9b42-4103-8dc7-e6b293264225" containerName="console"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046811    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-thanos"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046828    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d00f560-9b42-4103-8dc7-e6b293264225" containerName="console"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046839    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046856    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="092dd973-6986-449b-88ae-665ee06426d8" containerName="console"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046866    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="config-reloader"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046877    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="prometheus"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046892    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="thanos-sidecar"
Sep 05 02:21:18.047135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.046901    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" containerName="kube-rbac-proxy-web"
Sep 05 02:21:18.048986 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.048960    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.048986 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.048985    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.049262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049242    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.049262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049262    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.049487 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049469    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.049543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049487    2560 scope.go:117] "RemoveContainer" containerID="3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"
Sep 05 02:21:18.049773 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049727    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e"} err="failed to get container status \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": rpc error: code = NotFound desc = could not find container \"3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e\": container with ID starting with 3f8a87674ac95e927cdb5db6cc235e47320e743ce75d61e2e793cdec2035971e not found: ID does not exist"
Sep 05 02:21:18.049773 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.049768    2560 scope.go:117] "RemoveContainer" containerID="f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"
Sep 05 02:21:18.050024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050001    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952"} err="failed to get container status \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": rpc error: code = NotFound desc = could not find container \"f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952\": container with ID starting with f6520dab7f145e1b12b097d5774656cdadf1cd9757633b26cc29dc8551933952 not found: ID does not exist"
Sep 05 02:21:18.050080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050025    2560 scope.go:117] "RemoveContainer" containerID="c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"
Sep 05 02:21:18.050280 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050257    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f"} err="failed to get container status \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": rpc error: code = NotFound desc = could not find container \"c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f\": container with ID starting with c1ea72a445dc3cf504e95a593513433042daf9c3fa08a7475ae2eb09cbf33d5f not found: ID does not exist"
Sep 05 02:21:18.050333 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050282    2560 scope.go:117] "RemoveContainer" containerID="68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"
Sep 05 02:21:18.050527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050508    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2"} err="failed to get container status \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": rpc error: code = NotFound desc = could not find container \"68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2\": container with ID starting with 68151e28c77ffb96388a1a79e420da45b49e7fdfac16737e2479dee2062844c2 not found: ID does not exist"
Sep 05 02:21:18.050647 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050528    2560 scope.go:117] "RemoveContainer" containerID="5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"
Sep 05 02:21:18.050799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050782    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1"} err="failed to get container status \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": rpc error: code = NotFound desc = could not find container \"5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1\": container with ID starting with 5f0abb8cb58a267cfdf624e1ddb6c5851033f05f8e10fcfc38d4a025cc493af1 not found: ID does not exist"
Sep 05 02:21:18.050849 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050800    2560 scope.go:117] "RemoveContainer" containerID="1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"
Sep 05 02:21:18.051003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.050987    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0"} err="failed to get container status \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": rpc error: code = NotFound desc = could not find container \"1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0\": container with ID starting with 1b1a16b4c47aa316a887336ff5df9afa9a0b714250d09ff7fd176aba47ce67d0 not found: ID does not exist"
Sep 05 02:21:18.051180 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.051003    2560 scope.go:117] "RemoveContainer" containerID="4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"
Sep 05 02:21:18.051240 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.051212    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e"} err="failed to get container status \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": rpc error: code = NotFound desc = could not find container \"4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e\": container with ID starting with 4ae6cd3b26bbc2951dce12f9d3ce85d4aa4ecddf382708eb543ac4f0280f149e not found: ID does not exist"
Sep 05 02:21:18.051692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.051676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.054408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.054373    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 05 02:21:18.054935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.054913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 05 02:21:18.054935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.054927    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 05 02:21:18.054935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.054913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 05 02:21:18.055205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055033    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 05 02:21:18.055205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055181    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9p5c023envmqm\""
Sep 05 02:21:18.055205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 05 02:21:18.055383 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055220    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rcx7c\""
Sep 05 02:21:18.055449 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055382    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 05 02:21:18.055631 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055573    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 05 02:21:18.055767 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 05 02:21:18.055767 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055643    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 05 02:21:18.055767 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 05 02:21:18.055767 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.055721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 05 02:21:18.057502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.057483    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 05 02:21:18.060697 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.060677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 05 02:21:18.067943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.067923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:18.192390 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.192674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192714    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m66v\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-kube-api-access-5m66v\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.193116 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.192983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293546 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293546 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293546 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m66v\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-kube-api-access-5m66v\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.293732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.293685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294727 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.294916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.295110 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.295110 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.294966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.295110 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.295006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.295657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.295632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.296416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.296320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.296507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.296487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.297099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.296689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.297099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.296767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.297099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.297031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.297163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.297184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.297625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.297678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.298133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.298154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.298520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.298181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1b88099-048d-4ad6-8cf6-91b5a417031b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.299222 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.299201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.299284 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.299202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.299533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.299517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.299850 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.299834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1b88099-048d-4ad6-8cf6-91b5a417031b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.302522 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.302503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m66v\" (UniqueName: \"kubernetes.io/projected/a1b88099-048d-4ad6-8cf6-91b5a417031b-kube-api-access-5m66v\") pod \"prometheus-k8s-0\" (UID: \"a1b88099-048d-4ad6-8cf6-91b5a417031b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.361090 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.361055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:18.644140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.644113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:21:18.644817 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:21:18.644793    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b88099_048d_4ad6_8cf6_91b5a417031b.slice/crio-509f57194365b660ff72645251ae50a789bc2cf159d23db88263002f7fcce37c WatchSource:0}: Error finding container 509f57194365b660ff72645251ae50a789bc2cf159d23db88263002f7fcce37c: Status 404 returned error can't find the container with id 509f57194365b660ff72645251ae50a789bc2cf159d23db88263002f7fcce37c
Sep 05 02:21:18.972571 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.972500    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e215d34f-4bfd-4d9d-9d69-18860a39e8c9" path="/var/lib/kubelet/pods/e215d34f-4bfd-4d9d-9d69-18860a39e8c9/volumes"
Sep 05 02:21:18.990034 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.990006    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1b88099-048d-4ad6-8cf6-91b5a417031b" containerID="218717e9d17ba978f22cdc0ce71ea1528b78b0950776c3b3e81164bac558a7f0" exitCode=0
Sep 05 02:21:18.990149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.990068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerDied","Data":"218717e9d17ba978f22cdc0ce71ea1528b78b0950776c3b3e81164bac558a7f0"}
Sep 05 02:21:18.990149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:18.990119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"509f57194365b660ff72645251ae50a789bc2cf159d23db88263002f7fcce37c"}
Sep 05 02:21:19.621327 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.621258    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:21:19.623406 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.623390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.636367 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.636343    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:21:19.709402 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.709713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.709640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fll2\" (UniqueName: \"kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810647 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fll2\" (UniqueName: \"kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810878 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810878 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.810973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.810913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.811441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.811413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.811603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.811570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.811603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.811594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.811730 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.811686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.813388 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.813354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.813515 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.813492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.819298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.819274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fll2\" (UniqueName: \"kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2\") pod \"console-7fc7cc8857-92w6h\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") " pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.931555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.931526    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:19.999379 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"d5e2f4ae51ea78cacdc77f0f10d35557caf5d435ec1c3665e1ca453685a55a4c"}
Sep 05 02:21:19.999535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"415b81f27bf9a6d95d434eafa319001d31a1549bc85562007bf6cf91bbc62bcf"}
Sep 05 02:21:19.999535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"449242c2f3a0558600c4ab059bce9fbaa2a56a75b2c97f3eaad79b37fe00d233"}
Sep 05 02:21:19.999535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"1b88de9e77ab2561b49e9dea1ab21537acd0a0891cfddd51dca0a10ae1055963"}
Sep 05 02:21:19.999535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"a73cb2894ea61100a48f624aacbec564cd8392fbd1a775818ffd65aa223b89d3"}
Sep 05 02:21:19.999535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:19.999521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1b88099-048d-4ad6-8cf6-91b5a417031b","Type":"ContainerStarted","Data":"8837a895d27374310ef3b9460c4c45ac0143c38ffa4290b93bba345d8afcf4bb"}
Sep 05 02:21:20.036098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:20.036021    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.014447575 podStartE2EDuration="2.03600656s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.558985ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:20.034737037 +0000 UTC m=+223.673948887" watchObservedRunningTime="2026-09-05 02:21:20.03600656 +0000 UTC m=+223.675218402"
Sep 05 02:21:20.199425 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:20.199398    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:21:20.201785 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:21:20.201763    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0011ef_f8f2_4075_b941_27cfb15b36db.slice/crio-65ffcb5ae066443eda9b033af69f27256071f57c2130be7d7d3cbb8ad17f69b5 WatchSource:0}: Error finding container 65ffcb5ae066443eda9b033af69f27256071f57c2130be7d7d3cbb8ad17f69b5: Status 404 returned error can't find the container with id 65ffcb5ae066443eda9b033af69f27256071f57c2130be7d7d3cbb8ad17f69b5
Sep 05 02:21:21.004705 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:21.004670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc7cc8857-92w6h" event={"ID":"3c0011ef-f8f2-4075-b941-27cfb15b36db","Type":"ContainerStarted","Data":"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"}
Sep 05 02:21:21.005382 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:21.004726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc7cc8857-92w6h" event={"ID":"3c0011ef-f8f2-4075-b941-27cfb15b36db","Type":"ContainerStarted","Data":"65ffcb5ae066443eda9b033af69f27256071f57c2130be7d7d3cbb8ad17f69b5"}
Sep 05 02:21:21.024115 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:21.024066    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fc7cc8857-92w6h" podStartSLOduration=2.024044748 podStartE2EDuration="2.024044748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:21.022530499 +0000 UTC m=+224.661742318" watchObservedRunningTime="2026-09-05 02:21:21.024044748 +0000 UTC m=+224.663256622"
Sep 05 02:21:23.361784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:23.361731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:29.932158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:29.932113    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:29.932543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:29.932187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:29.936833 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:29.936805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:30.043863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:30.043836    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:21:30.092810 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:30.092782    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:21:55.118344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.118283    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765f6c74db-x565l" podUID="94f846dc-5443-4012-abf5-06b1b5b248df" containerName="console" containerID="cri-o://6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5" gracePeriod=15
Sep 05 02:21:55.349597 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.349570    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:21:55.371027 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.370974    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371027 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371056    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371110    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371133    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8k2x\" (UniqueName: \"kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config\") pod \"94f846dc-5443-4012-abf5-06b1b5b248df\" (UID: \"94f846dc-5443-4012-abf5-06b1b5b248df\") "
Sep 05 02:21:55.371567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:55.371686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config" (OuterVolumeSpecName: "console-config") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:55.371791 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371764    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:55.371865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.371681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca" (OuterVolumeSpecName: "service-ca") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:55.373564 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.373529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:55.373683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.373637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:55.373683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.373655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x" (OuterVolumeSpecName: "kube-api-access-w8k2x") pod "94f846dc-5443-4012-abf5-06b1b5b248df" (UID: "94f846dc-5443-4012-abf5-06b1b5b248df"). InnerVolumeSpecName "kube-api-access-w8k2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:55.472211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472177    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-service-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472210    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-trusted-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472220    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-oauth-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472229    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94f846dc-5443-4012-abf5-06b1b5b248df-console-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472238    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472245    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8k2x\" (UniqueName: \"kubernetes.io/projected/94f846dc-5443-4012-abf5-06b1b5b248df-kube-api-access-w8k2x\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:55.472373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:55.472254    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94f846dc-5443-4012-abf5-06b1b5b248df-console-oauth-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:21:56.143011 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.142975    2560 generic.go:356] "Generic (PLEG): container finished" podID="94f846dc-5443-4012-abf5-06b1b5b248df" containerID="6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5" exitCode=0
Sep 05 02:21:56.143393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.143016    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f6c74db-x565l"
Sep 05 02:21:56.143393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.143057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f6c74db-x565l" event={"ID":"94f846dc-5443-4012-abf5-06b1b5b248df","Type":"ContainerDied","Data":"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"}
Sep 05 02:21:56.143393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.143111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f6c74db-x565l" event={"ID":"94f846dc-5443-4012-abf5-06b1b5b248df","Type":"ContainerDied","Data":"e090865f4c4fbd2aff1d1b7b2a8b169d0b4aca20db2e56710da3774d3ffe2e6a"}
Sep 05 02:21:56.143393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.143132    2560 scope.go:117] "RemoveContainer" containerID="6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"
Sep 05 02:21:56.153652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.153617    2560 scope.go:117] "RemoveContainer" containerID="6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"
Sep 05 02:21:56.153971 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:21:56.153946    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5\": container with ID starting with 6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5 not found: ID does not exist" containerID="6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"
Sep 05 02:21:56.154047 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.153977    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5"} err="failed to get container status \"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5\": rpc error: code = NotFound desc = could not find container \"6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5\": container with ID starting with 6e5bb2e2ffb1793ea78a8716c4186a6f70bb9c671c80de2717da52bd065bdfe5 not found: ID does not exist"
Sep 05 02:21:56.166029 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.166005    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:21:56.169817 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.169794    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765f6c74db-x565l"]
Sep 05 02:21:56.969756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:21:56.969711    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f846dc-5443-4012-abf5-06b1b5b248df" path="/var/lib/kubelet/pods/94f846dc-5443-4012-abf5-06b1b5b248df/volumes"
Sep 05 02:22:18.362140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:18.362105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:18.378080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:18.378057    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:19.250409 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:19.250379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:30.407035 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.406996    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:22:30.407613 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.407475    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f846dc-5443-4012-abf5-06b1b5b248df" containerName="console"
Sep 05 02:22:30.407613 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.407495    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f846dc-5443-4012-abf5-06b1b5b248df" containerName="console"
Sep 05 02:22:30.407613 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.407582    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f846dc-5443-4012-abf5-06b1b5b248df" containerName="console"
Sep 05 02:22:30.410580 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.410554    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.424656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.424633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:22:30.532321 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246f5\" (UniqueName: \"kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.532620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.532561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633083 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633083 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246f5\" (UniqueName: \"kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.633937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.634066 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.633934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.634066 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.634014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.635589 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.635569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.635798 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.635780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.641035 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.641011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246f5\" (UniqueName: \"kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5\") pod \"console-966c496c9-rr5ff\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") " pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.719933 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.719863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:30.994154 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:30.994132    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:22:30.996530 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:22:30.996503    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aceaf27_a1f5_488c_9091_f1aa9cb02205.slice/crio-1b51c86c3a26f0b22b3f8b4e864943cc6ce9b5c5c3a433b785d55fdff31a647b WatchSource:0}: Error finding container 1b51c86c3a26f0b22b3f8b4e864943cc6ce9b5c5c3a433b785d55fdff31a647b: Status 404 returned error can't find the container with id 1b51c86c3a26f0b22b3f8b4e864943cc6ce9b5c5c3a433b785d55fdff31a647b
Sep 05 02:22:31.287177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:31.287097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966c496c9-rr5ff" event={"ID":"4aceaf27-a1f5-488c-9091-f1aa9cb02205","Type":"ContainerStarted","Data":"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"}
Sep 05 02:22:31.287177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:31.287146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966c496c9-rr5ff" event={"ID":"4aceaf27-a1f5-488c-9091-f1aa9cb02205","Type":"ContainerStarted","Data":"1b51c86c3a26f0b22b3f8b4e864943cc6ce9b5c5c3a433b785d55fdff31a647b"}
Sep 05 02:22:31.305161 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:31.305111    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-966c496c9-rr5ff" podStartSLOduration=1.305096289 podStartE2EDuration="1.305096289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:31.304779304 +0000 UTC m=+294.943991124" watchObservedRunningTime="2026-09-05 02:22:31.305096289 +0000 UTC m=+294.944308149"
Sep 05 02:22:36.743596 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:36.743566    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:22:36.744678 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:36.744650    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:22:36.747602 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:36.747580    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:22:40.720610 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:40.720562    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:40.723190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:40.720631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:40.725358 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:40.725339    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:41.329907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:41.329871    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:22:41.373121 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:22:41.373093    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:23:06.392377 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.392314    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fc7cc8857-92w6h" podUID="3c0011ef-f8f2-4075-b941-27cfb15b36db" containerName="console" containerID="cri-o://1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac" gracePeriod=15
Sep 05 02:23:06.629849 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.629826    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:23:06.808182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808116    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808172    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fll2\" (UniqueName: \"kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808276    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca\") pod \"3c0011ef-f8f2-4075-b941-27cfb15b36db\" (UID: \"3c0011ef-f8f2-4075-b941-27cfb15b36db\") "
Sep 05 02:23:06.808646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:06.808731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config" (OuterVolumeSpecName: "console-config") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:06.808814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:06.808864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808848    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-trusted-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.808915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808872    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.808915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.808886    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-oauth-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.809049 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.809031    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca" (OuterVolumeSpecName: "service-ca") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:06.810431 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.810407    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:23:06.810836 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.810808    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2" (OuterVolumeSpecName: "kube-api-access-5fll2") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "kube-api-access-5fll2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:23:06.810836 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.810805    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3c0011ef-f8f2-4075-b941-27cfb15b36db" (UID: "3c0011ef-f8f2-4075-b941-27cfb15b36db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:23:06.909251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.909226    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fll2\" (UniqueName: \"kubernetes.io/projected/3c0011ef-f8f2-4075-b941-27cfb15b36db-kube-api-access-5fll2\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.909251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.909247    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-oauth-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.909376 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.909257    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0011ef-f8f2-4075-b941-27cfb15b36db-console-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:06.909376 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:06.909266    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0011ef-f8f2-4075-b941-27cfb15b36db-service-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:23:07.426917 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.426881    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c0011ef-f8f2-4075-b941-27cfb15b36db" containerID="1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac" exitCode=0
Sep 05 02:23:07.427287 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.426921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc7cc8857-92w6h" event={"ID":"3c0011ef-f8f2-4075-b941-27cfb15b36db","Type":"ContainerDied","Data":"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"}
Sep 05 02:23:07.427287 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.426929    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc7cc8857-92w6h"
Sep 05 02:23:07.427287 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.426979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc7cc8857-92w6h" event={"ID":"3c0011ef-f8f2-4075-b941-27cfb15b36db","Type":"ContainerDied","Data":"65ffcb5ae066443eda9b033af69f27256071f57c2130be7d7d3cbb8ad17f69b5"}
Sep 05 02:23:07.427287 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.427009    2560 scope.go:117] "RemoveContainer" containerID="1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"
Sep 05 02:23:07.435423 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.435393    2560 scope.go:117] "RemoveContainer" containerID="1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"
Sep 05 02:23:07.435710 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:23:07.435686    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac\": container with ID starting with 1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac not found: ID does not exist" containerID="1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"
Sep 05 02:23:07.435784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.435719    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac"} err="failed to get container status \"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac\": rpc error: code = NotFound desc = could not find container \"1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac\": container with ID starting with 1761dd3113e60585efaac37bae49d1ccea5d6cffd27d1255457785ed0e695cac not found: ID does not exist"
Sep 05 02:23:07.442881 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.442856    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:23:07.446316 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:07.446294    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fc7cc8857-92w6h"]
Sep 05 02:23:08.974685 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:23:08.974648    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0011ef-f8f2-4075-b941-27cfb15b36db" path="/var/lib/kubelet/pods/3c0011ef-f8f2-4075-b941-27cfb15b36db/volumes"
Sep 05 02:25:09.614312 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.614238    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59fd47449d-q5dcq"]
Sep 05 02:25:09.614679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.614555    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0011ef-f8f2-4075-b941-27cfb15b36db" containerName="console"
Sep 05 02:25:09.614679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.614567    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0011ef-f8f2-4075-b941-27cfb15b36db" containerName="console"
Sep 05 02:25:09.614679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.614630    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0011ef-f8f2-4075-b941-27cfb15b36db" containerName="console"
Sep 05 02:25:09.617180 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.617163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.628618 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.628594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fd47449d-q5dcq"]
Sep 05 02:25:09.681927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.681902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-oauth-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.681941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-console-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.682000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.682023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-oauth-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.682043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhls4\" (UniqueName: \"kubernetes.io/projected/124dd00b-9787-4a17-aa02-142fce353e8a-kube-api-access-lhls4\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.682097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-trusted-ca-bundle\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.682158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.682127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-service-ca\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.782817 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-oauth-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.782903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhls4\" (UniqueName: \"kubernetes.io/projected/124dd00b-9787-4a17-aa02-142fce353e8a-kube-api-access-lhls4\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.782903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-trusted-ca-bundle\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.782903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-service-ca\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-oauth-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.782944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-console-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783150 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.783120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783575 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.783552    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-oauth-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.783615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-console-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.783619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-service-ca\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.783767 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.783695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/124dd00b-9787-4a17-aa02-142fce353e8a-trusted-ca-bundle\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.785544 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.785518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-serving-cert\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.785637 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.785619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/124dd00b-9787-4a17-aa02-142fce353e8a-console-oauth-config\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.792137 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.792115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhls4\" (UniqueName: \"kubernetes.io/projected/124dd00b-9787-4a17-aa02-142fce353e8a-kube-api-access-lhls4\") pod \"console-59fd47449d-q5dcq\" (UID: \"124dd00b-9787-4a17-aa02-142fce353e8a\") " pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:09.926091 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:09.926070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:10.245610 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:10.245583    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fd47449d-q5dcq"]
Sep 05 02:25:10.247981 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:25:10.247950    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124dd00b_9787_4a17_aa02_142fce353e8a.slice/crio-3d3fb3657e43e6bce0c00ca5b4dc90f91d7d3ec127d755140c56029d38e207e7 WatchSource:0}: Error finding container 3d3fb3657e43e6bce0c00ca5b4dc90f91d7d3ec127d755140c56029d38e207e7: Status 404 returned error can't find the container with id 3d3fb3657e43e6bce0c00ca5b4dc90f91d7d3ec127d755140c56029d38e207e7
Sep 05 02:25:10.912056 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:10.912011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fd47449d-q5dcq" event={"ID":"124dd00b-9787-4a17-aa02-142fce353e8a","Type":"ContainerStarted","Data":"2795d19b2df1c21d7354f97015a6a093d4e5d8eb547ed36aedf897dfdb256834"}
Sep 05 02:25:10.912410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:10.912110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fd47449d-q5dcq" event={"ID":"124dd00b-9787-4a17-aa02-142fce353e8a","Type":"ContainerStarted","Data":"3d3fb3657e43e6bce0c00ca5b4dc90f91d7d3ec127d755140c56029d38e207e7"}
Sep 05 02:25:19.926590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:19.926544    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:19.927118 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:19.926617    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:19.931183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:19.931160    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:19.953258 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:19.953209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59fd47449d-q5dcq" podStartSLOduration=10.953194088 podStartE2EDuration="10.953194088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:10.931866593 +0000 UTC m=+454.571078412" watchObservedRunningTime="2026-09-05 02:25:19.953194088 +0000 UTC m=+463.592405887"
Sep 05 02:25:19.955455 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:19.955434    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59fd47449d-q5dcq"
Sep 05 02:25:20.006684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:20.006655    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:25:45.027926 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.027865    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-966c496c9-rr5ff" podUID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" containerName="console" containerID="cri-o://f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2" gracePeriod=15
Sep 05 02:25:45.267991 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.267968    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:25:45.364024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.363939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.363990    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364245 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364055    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-246f5\" (UniqueName: \"kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364245 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364085    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364245 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364245 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364162    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert\") pod \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\" (UID: \"4aceaf27-a1f5-488c-9091-f1aa9cb02205\") "
Sep 05 02:25:45.364412 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:45.364469 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca" (OuterVolumeSpecName: "service-ca") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:45.364770 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364708    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config" (OuterVolumeSpecName: "console-config") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:45.364770 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.364735    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:45.366271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.366241    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:25:45.366271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.366254    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:25:45.366432 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.366337    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5" (OuterVolumeSpecName: "kube-api-access-246f5") pod "4aceaf27-a1f5-488c-9091-f1aa9cb02205" (UID: "4aceaf27-a1f5-488c-9091-f1aa9cb02205"). InnerVolumeSpecName "kube-api-access-246f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:25:45.464793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464736    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464787    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464797    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-oauth-serving-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464806    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-trusted-ca-bundle\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464814    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aceaf27-a1f5-488c-9091-f1aa9cb02205-service-ca\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464823    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aceaf27-a1f5-488c-9091-f1aa9cb02205-console-oauth-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:45.464981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:45.464831    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246f5\" (UniqueName: \"kubernetes.io/projected/4aceaf27-a1f5-488c-9091-f1aa9cb02205-kube-api-access-246f5\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:25:46.050549 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.050521    2560 generic.go:356] "Generic (PLEG): container finished" podID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" containerID="f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2" exitCode=0
Sep 05 02:25:46.050934 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.050558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966c496c9-rr5ff" event={"ID":"4aceaf27-a1f5-488c-9091-f1aa9cb02205","Type":"ContainerDied","Data":"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"}
Sep 05 02:25:46.050934 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.050568    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966c496c9-rr5ff"
Sep 05 02:25:46.050934 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.050591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966c496c9-rr5ff" event={"ID":"4aceaf27-a1f5-488c-9091-f1aa9cb02205","Type":"ContainerDied","Data":"1b51c86c3a26f0b22b3f8b4e864943cc6ce9b5c5c3a433b785d55fdff31a647b"}
Sep 05 02:25:46.050934 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.050612    2560 scope.go:117] "RemoveContainer" containerID="f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"
Sep 05 02:25:46.061793 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.061764    2560 scope.go:117] "RemoveContainer" containerID="f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"
Sep 05 02:25:46.066104 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:25:46.066064    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2\": container with ID starting with f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2 not found: ID does not exist" containerID="f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"
Sep 05 02:25:46.066197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.066115    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2"} err="failed to get container status \"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2\": rpc error: code = NotFound desc = could not find container \"f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2\": container with ID starting with f04d0fe3c809b0ad5e5904e76a2be36aa108b4723262bb743f6de82ec74d7df2 not found: ID does not exist"
Sep 05 02:25:46.081683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.081657    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:25:46.086452 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.086433    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-966c496c9-rr5ff"]
Sep 05 02:25:46.971339 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:25:46.971303    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" path="/var/lib/kubelet/pods/4aceaf27-a1f5-488c-9091-f1aa9cb02205/volumes"
Sep 05 02:27:36.769822 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:36.769795    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:27:36.770347 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:36.770242    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:27:39.567488 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.567456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2fzbg"]
Sep 05 02:27:39.567860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.567775    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" containerName="console"
Sep 05 02:27:39.567860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.567786    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" containerName="console"
Sep 05 02:27:39.567860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.567847    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aceaf27-a1f5-488c-9091-f1aa9cb02205" containerName="console"
Sep 05 02:27:39.570616 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.570601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.572950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.572926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.573064 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.572961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvw9s\" (UniqueName: \"kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.573200 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.573180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:27:39.573314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.573186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:27:39.573314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.573247    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 05 02:27:39.573314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.573234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:27:39.574297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.574265    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rb75m\""
Sep 05 02:27:39.579762 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.579725    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2fzbg"]
Sep 05 02:27:39.673832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.673804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.673920 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.673839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvw9s\" (UniqueName: \"kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.674157 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.674134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.682822 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.682803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvw9s\" (UniqueName: \"kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s\") pod \"s3-init-2fzbg\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") " pod="kserve/s3-init-2fzbg"
Sep 05 02:27:39.890673 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:39.890656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2fzbg"
Sep 05 02:27:40.163248 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:40.161923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2fzbg"]
Sep 05 02:27:40.165579 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:27:40.165547    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf15b75c_2d31_4bdb_ae98_9968708590a2.slice/crio-0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c WatchSource:0}: Error finding container 0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c: Status 404 returned error can't find the container with id 0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c
Sep 05 02:27:40.167557 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:40.167537    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:27:40.487937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:40.487856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerStarted","Data":"0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c"}
Sep 05 02:27:44.509333 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:44.509291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerStarted","Data":"87265b0af49c14308fcc61243e95a47034e1e1378d736d96ffb5b8fc9a2e6fc3"}
Sep 05 02:27:45.514927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:45.514894    2560 generic.go:356] "Generic (PLEG): container finished" podID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerID="87265b0af49c14308fcc61243e95a47034e1e1378d736d96ffb5b8fc9a2e6fc3" exitCode=0
Sep 05 02:27:45.515295 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:45.514982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerDied","Data":"87265b0af49c14308fcc61243e95a47034e1e1378d736d96ffb5b8fc9a2e6fc3"}
Sep 05 02:27:50.540376 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:50.540337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerStarted","Data":"3fa25f9275adf1aed48d84ab0af3473f8bd7f7d9c883cc1755c360c35f787ee1"}
Sep 05 02:27:50.561067 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:27:50.561016    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2fzbg" podStartSLOduration=2.361128396 podStartE2EDuration="11.561002184s" totalImagesPullingTime="7.670152525s" totalInitContainerRuntime="1.529721263s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:50.559341703 +0000 UTC m=+614.198553505" watchObservedRunningTime="2026-09-05 02:27:50.561002184 +0000 UTC m=+614.200214110"
Sep 05 02:28:54.791959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:54.791924    2560 generic.go:356] "Generic (PLEG): container finished" podID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerID="3fa25f9275adf1aed48d84ab0af3473f8bd7f7d9c883cc1755c360c35f787ee1" exitCode=0
Sep 05 02:28:54.792310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:54.791954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerDied","Data":"3fa25f9275adf1aed48d84ab0af3473f8bd7f7d9c883cc1755c360c35f787ee1"}
Sep 05 02:28:55.922618 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:55.922595    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2fzbg"
Sep 05 02:28:55.976405 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:55.976380    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvw9s\" (UniqueName: \"kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s\") pod \"bf15b75c-2d31-4bdb-ae98-9968708590a2\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") "
Sep 05 02:28:55.976552 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:55.976419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache\") pod \"bf15b75c-2d31-4bdb-ae98-9968708590a2\" (UID: \"bf15b75c-2d31-4bdb-ae98-9968708590a2\") "
Sep 05 02:28:55.978505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:55.978468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s" (OuterVolumeSpecName: "kube-api-access-wvw9s") pod "bf15b75c-2d31-4bdb-ae98-9968708590a2" (UID: "bf15b75c-2d31-4bdb-ae98-9968708590a2"). InnerVolumeSpecName "kube-api-access-wvw9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:55.995001 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:55.994977    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf15b75c-2d31-4bdb-ae98-9968708590a2" (UID: "bf15b75c-2d31-4bdb-ae98-9968708590a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:28:56.077495 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:56.077429    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvw9s\" (UniqueName: \"kubernetes.io/projected/bf15b75c-2d31-4bdb-ae98-9968708590a2-kube-api-access-wvw9s\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:28:56.077495 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:56.077459    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf15b75c-2d31-4bdb-ae98-9968708590a2-model-cache\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:28:56.802199 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:56.802176    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2fzbg"
Sep 05 02:28:56.802353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:56.802206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2fzbg" event={"ID":"bf15b75c-2d31-4bdb-ae98-9968708590a2","Type":"ContainerDied","Data":"0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c"}
Sep 05 02:28:56.802353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:28:56.802236    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9f180091f35389858fbe42db861cb15ad2d132cfe6ea6206cb765b13386e8c"
Sep 05 02:29:15.580555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580520    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"]
Sep 05 02:29:15.580949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580849    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerName="download-hf-model"
Sep 05 02:29:15.580949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580861    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerName="download-hf-model"
Sep 05 02:29:15.580949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580870    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerName="s3-init"
Sep 05 02:29:15.580949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580875    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerName="s3-init"
Sep 05 02:29:15.580949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.580923    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf15b75c-2d31-4bdb-ae98-9968708590a2" containerName="s3-init"
Sep 05 02:29:15.583615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.583600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.586298 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.586278    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:29:15.587355 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.587334    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:29:15.587355 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.587348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 05 02:29:15.587506 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.587341    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 05 02:29:15.587506 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.587439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rb75m\""
Sep 05 02:29:15.592522 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.592500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"]
Sep 05 02:29:15.715709 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.715680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rwj\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-kube-api-access-d8rwj\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.715864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.715759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95e83c31-5be6-4897-9482-f68df1952b45-data\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.715864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.715822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.816874 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.816846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.816995 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.816880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rwj\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-kube-api-access-d8rwj\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.817077 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.817057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95e83c31-5be6-4897-9482-f68df1952b45-data\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.817363 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.817346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95e83c31-5be6-4897-9482-f68df1952b45-data\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.819372 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.819350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.825088 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.825068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rwj\" (UniqueName: \"kubernetes.io/projected/95e83c31-5be6-4897-9482-f68df1952b45-kube-api-access-d8rwj\") pod \"seaweedfs-tls-custom-8b44d767d-k6ftj\" (UID: \"95e83c31-5be6-4897-9482-f68df1952b45\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:15.893494 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:15.893474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"
Sep 05 02:29:16.159960 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:16.159934    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k6ftj"]
Sep 05 02:29:16.162319 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:29:16.162291    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e83c31_5be6_4897_9482_f68df1952b45.slice/crio-e717daba18d8ab900254a041e0dd98eea7746480c95173738fc89401a4109fc8 WatchSource:0}: Error finding container e717daba18d8ab900254a041e0dd98eea7746480c95173738fc89401a4109fc8: Status 404 returned error can't find the container with id e717daba18d8ab900254a041e0dd98eea7746480c95173738fc89401a4109fc8
Sep 05 02:29:16.886476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:16.886442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj" event={"ID":"95e83c31-5be6-4897-9482-f68df1952b45","Type":"ContainerStarted","Data":"e717daba18d8ab900254a041e0dd98eea7746480c95173738fc89401a4109fc8"}
Sep 05 02:29:18.897140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:18.897099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj" event={"ID":"95e83c31-5be6-4897-9482-f68df1952b45","Type":"ContainerStarted","Data":"bde90d4ae818238e58f69eb53b0ad254e369407de760105534d97c2605d5d709"}
Sep 05 02:29:18.914431 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:18.914372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-k6ftj" podStartSLOduration=1.295235403 podStartE2EDuration="3.914356625s" totalImagesPullingTime="2.619121222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:18.913041363 +0000 UTC m=+702.552253182" watchObservedRunningTime="2026-09-05 02:29:18.914356625 +0000 UTC m=+702.553568422"
Sep 05 02:29:19.206302 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.206269    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-mh47x"]
Sep 05 02:29:19.209449 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.209432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.211865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.211844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:29:19.218171 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.218151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-mh47x"]
Sep 05 02:29:19.347463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.347428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkz9d\" (UniqueName: \"kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.347463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.347468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.447950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.447916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.448096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.448025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkz9d\" (UniqueName: \"kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.448265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.448244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.456801 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.456730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkz9d\" (UniqueName: \"kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d\") pod \"s3-tls-init-custom-mh47x\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") " pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.531574 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.531543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:19.801043 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.800975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-mh47x"]
Sep 05 02:29:19.804304 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:29:19.804275    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462ac25e_9aa3_416a_994d_27b724837786.slice/crio-e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04 WatchSource:0}: Error finding container e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04: Status 404 returned error can't find the container with id e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04
Sep 05 02:29:19.902414 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.902384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerStarted","Data":"33e1213176467719e6f00764fdeda39059739647ca6a76483a55365ca7dbb0b0"}
Sep 05 02:29:19.902753 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:19.902427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerStarted","Data":"e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04"}
Sep 05 02:29:21.912873 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:21.912841    2560 generic.go:356] "Generic (PLEG): container finished" podID="462ac25e-9aa3-416a-994d-27b724837786" containerID="33e1213176467719e6f00764fdeda39059739647ca6a76483a55365ca7dbb0b0" exitCode=0
Sep 05 02:29:21.913224 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:21.912913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerDied","Data":"33e1213176467719e6f00764fdeda39059739647ca6a76483a55365ca7dbb0b0"}
Sep 05 02:29:22.919061 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:22.919025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerStarted","Data":"719238a69411778e4ce942193238ebf40f82dda2bf119fdc782c527f12a37fa5"}
Sep 05 02:29:22.935325 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:22.935283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-mh47x" podStartSLOduration=2.528338807 podStartE2EDuration="3.935268269s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.406929462s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:22.934156719 +0000 UTC m=+706.573368541" watchObservedRunningTime="2026-09-05 02:29:22.935268269 +0000 UTC m=+706.574480069"
Sep 05 02:29:27.943128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:27.943093    2560 generic.go:356] "Generic (PLEG): container finished" podID="462ac25e-9aa3-416a-994d-27b724837786" containerID="719238a69411778e4ce942193238ebf40f82dda2bf119fdc782c527f12a37fa5" exitCode=0
Sep 05 02:29:27.943567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:27.943145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerDied","Data":"719238a69411778e4ce942193238ebf40f82dda2bf119fdc782c527f12a37fa5"}
Sep 05 02:29:29.084651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.084628    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:29.234381 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.234319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkz9d\" (UniqueName: \"kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d\") pod \"462ac25e-9aa3-416a-994d-27b724837786\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") "
Sep 05 02:29:29.234517 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.234424    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache\") pod \"462ac25e-9aa3-416a-994d-27b724837786\" (UID: \"462ac25e-9aa3-416a-994d-27b724837786\") "
Sep 05 02:29:29.236445 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.236410    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d" (OuterVolumeSpecName: "kube-api-access-tkz9d") pod "462ac25e-9aa3-416a-994d-27b724837786" (UID: "462ac25e-9aa3-416a-994d-27b724837786"). InnerVolumeSpecName "kube-api-access-tkz9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:29:29.255850 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.255821    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache" (OuterVolumeSpecName: "model-cache") pod "462ac25e-9aa3-416a-994d-27b724837786" (UID: "462ac25e-9aa3-416a-994d-27b724837786"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:29:29.335659 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.335635    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462ac25e-9aa3-416a-994d-27b724837786-model-cache\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:29:29.335659 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.335656    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkz9d\" (UniqueName: \"kubernetes.io/projected/462ac25e-9aa3-416a-994d-27b724837786-kube-api-access-tkz9d\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:29:29.952461 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.952430    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mh47x"
Sep 05 02:29:29.952628 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.952465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mh47x" event={"ID":"462ac25e-9aa3-416a-994d-27b724837786","Type":"ContainerDied","Data":"e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04"}
Sep 05 02:29:29.952628 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:29.952503    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a17de8d1f7161c74408d341b143ff50700cb691612afe9c3ca6698feb08b04"
Sep 05 02:29:33.151108 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151075    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-jt2hp"]
Sep 05 02:29:33.151477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151370    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462ac25e-9aa3-416a-994d-27b724837786" containerName="s3-tls-init-custom"
Sep 05 02:29:33.151477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151381    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="462ac25e-9aa3-416a-994d-27b724837786" containerName="s3-tls-init-custom"
Sep 05 02:29:33.151477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151404    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462ac25e-9aa3-416a-994d-27b724837786" containerName="download-hf-model"
Sep 05 02:29:33.151477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151410    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="462ac25e-9aa3-416a-994d-27b724837786" containerName="download-hf-model"
Sep 05 02:29:33.151477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.151462    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="462ac25e-9aa3-416a-994d-27b724837786" containerName="s3-tls-init-custom"
Sep 05 02:29:33.154349 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.154334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.156689 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.156665    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:29:33.156943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.156924    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 05 02:29:33.162240 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.162208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-jt2hp"]
Sep 05 02:29:33.263408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.263384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548vh\" (UniqueName: \"kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.263504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.263435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.364091 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.364067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548vh\" (UniqueName: \"kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.364205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.364144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.364512 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.364491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.374357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.374331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-548vh\" (UniqueName: \"kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh\") pod \"s3-tls-init-serving-jt2hp\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") " pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.474329 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.474279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:33.742881 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.742857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-jt2hp"]
Sep 05 02:29:33.745019 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:29:33.744989    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cf9d1cc_5907_448c_abe4_f837c7aafc16.slice/crio-f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63 WatchSource:0}: Error finding container f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63: Status 404 returned error can't find the container with id f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63
Sep 05 02:29:33.968920 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.968876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerStarted","Data":"15b6c96c4dba49fb8efbb79bb241ed1d10eca88aa21d491eecbf22a5b5a7bb90"}
Sep 05 02:29:33.969068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:33.968933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerStarted","Data":"f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63"}
Sep 05 02:29:35.978984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:35.978954    2560 generic.go:356] "Generic (PLEG): container finished" podID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerID="15b6c96c4dba49fb8efbb79bb241ed1d10eca88aa21d491eecbf22a5b5a7bb90" exitCode=0
Sep 05 02:29:35.979372 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:35.979041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerDied","Data":"15b6c96c4dba49fb8efbb79bb241ed1d10eca88aa21d491eecbf22a5b5a7bb90"}
Sep 05 02:29:36.984703 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:36.984667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerStarted","Data":"a32e3b5d3e7b00079628c6dd7e50613a446de99cad4e332ce40082112ac107ec"}
Sep 05 02:29:37.002953 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:37.002902    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-jt2hp" podStartSLOduration=2.551622888 podStartE2EDuration="4.002888337s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.451265449s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:37.001870471 +0000 UTC m=+720.641082273" watchObservedRunningTime="2026-09-05 02:29:37.002888337 +0000 UTC m=+720.642100154"
Sep 05 02:29:43.010880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:43.010847    2560 generic.go:356] "Generic (PLEG): container finished" podID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerID="a32e3b5d3e7b00079628c6dd7e50613a446de99cad4e332ce40082112ac107ec" exitCode=0
Sep 05 02:29:43.011258 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:43.010933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerDied","Data":"a32e3b5d3e7b00079628c6dd7e50613a446de99cad4e332ce40082112ac107ec"}
Sep 05 02:29:44.149040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.149018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:44.152291 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.152251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-548vh\" (UniqueName: \"kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh\") pod \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") "
Sep 05 02:29:44.152383 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.152368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache\") pod \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\" (UID: \"9cf9d1cc-5907-448c-abe4-f837c7aafc16\") "
Sep 05 02:29:44.154114 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.154088    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh" (OuterVolumeSpecName: "kube-api-access-548vh") pod "9cf9d1cc-5907-448c-abe4-f837c7aafc16" (UID: "9cf9d1cc-5907-448c-abe4-f837c7aafc16"). InnerVolumeSpecName "kube-api-access-548vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:29:44.175181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.175153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cf9d1cc-5907-448c-abe4-f837c7aafc16" (UID: "9cf9d1cc-5907-448c-abe4-f837c7aafc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:29:44.253391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.253368    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf9d1cc-5907-448c-abe4-f837c7aafc16-model-cache\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:29:44.253391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:44.253391    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-548vh\" (UniqueName: \"kubernetes.io/projected/9cf9d1cc-5907-448c-abe4-f837c7aafc16-kube-api-access-548vh\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:29:45.021174 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:45.021147    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jt2hp"
Sep 05 02:29:45.021314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:45.021188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jt2hp" event={"ID":"9cf9d1cc-5907-448c-abe4-f837c7aafc16","Type":"ContainerDied","Data":"f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63"}
Sep 05 02:29:45.021314 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:45.021226    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0884858c9b119d74bd8a76c2a6f3ad2ee3c9027a522edbefc348714e8d27a63"
Sep 05 02:29:57.587330 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587297    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:29:57.587808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587606    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerName="s3-tls-init-serving"
Sep 05 02:29:57.587808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587617    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerName="s3-tls-init-serving"
Sep 05 02:29:57.587808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587633    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerName="download-hf-model"
Sep 05 02:29:57.587808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587639    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerName="download-hf-model"
Sep 05 02:29:57.587808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.587688    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf9d1cc-5907-448c-abe4-f837c7aafc16" containerName="s3-tls-init-serving"
Sep 05 02:29:57.590128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.590111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.593214 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.593187    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:29:57.593335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.593215    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:29:57.593614 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.593597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-827065-dockercfg-8kfzn\""
Sep 05 02:29:57.594522 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.594503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-827065-predictor-serving-cert\""
Sep 05 02:29:57.594856 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.594838    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-827065-kube-rbac-proxy-sar-config\""
Sep 05 02:29:57.594922 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.594911    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:29:57.595059 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.595041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-827065\""
Sep 05 02:29:57.608404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.608383    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:29:57.642163 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.642129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.642257 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.642172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.642257 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.642207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.642338 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.642285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26tf\" (UniqueName: \"kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.642378 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.642356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26tf\" (UniqueName: \"kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743840 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.743840 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:29:57.743717    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-827065-predictor-serving-cert: secret "isvc-init-fail-827065-predictor-serving-cert" not found
Sep 05 02:29:57.743840 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:29:57.743795    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls podName:2f289988-287c-485c-b78f-d51abc06bac1 nodeName:}" failed. No retries permitted until 2026-09-05 02:29:58.243774754 +0000 UTC m=+741.882986546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls") pod "isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" (UID: "2f289988-287c-485c-b78f-d51abc06bac1") : secret "isvc-init-fail-827065-predictor-serving-cert" not found
Sep 05 02:29:57.743991 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.743929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.744553 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.744536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.744604 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.744584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.754645 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.754622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26tf\" (UniqueName: \"kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:57.801693 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.801662    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:29:57.804231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.804212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.807225 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.807202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:29:57.807329 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.807313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:29:57.807381 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.807348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 05 02:29:57.807780 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.807765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 05 02:29:57.820758 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.820723    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:29:57.844227 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.844166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.844227 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.844203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.844355 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.844228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.844355 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.844256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.844435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.844379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngrsb\" (UniqueName: \"kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.944981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945139 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrsb\" (UniqueName: \"kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945305 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945544 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.945837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.945818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.947507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.947485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:57.953522 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:57.953504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrsb\" (UniqueName: \"kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb\") pod \"isvc-sklearn-batcher-predictor-7b79d4588c-22rzb\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:58.114924 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.114859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:29:58.247923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.247882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:58.250142 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.250116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") pod \"isvc-init-fail-827065-predictor-69c84ddc55-8wwcw\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") " pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:58.386969 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.386941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:29:58.389727 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:29:58.389701    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd491b1eb_7607_4033_ac88_2138a2993925.slice/crio-81334220ecabd2882b05f8da017d52c375221e8279396f1d590162a7e640c0e8 WatchSource:0}: Error finding container 81334220ecabd2882b05f8da017d52c375221e8279396f1d590162a7e640c0e8: Status 404 returned error can't find the container with id 81334220ecabd2882b05f8da017d52c375221e8279396f1d590162a7e640c0e8
Sep 05 02:29:58.500502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.500474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:29:58.769413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:58.769376    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:29:58.771799 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:29:58.771776    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f289988_287c_485c_b78f_d51abc06bac1.slice/crio-98c428339094bfca495584f9c6567f0ee9d170277838858fbba32827fbd9a5f9 WatchSource:0}: Error finding container 98c428339094bfca495584f9c6567f0ee9d170277838858fbba32827fbd9a5f9: Status 404 returned error can't find the container with id 98c428339094bfca495584f9c6567f0ee9d170277838858fbba32827fbd9a5f9
Sep 05 02:29:59.080186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:59.080076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerStarted","Data":"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"}
Sep 05 02:29:59.080186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:59.080170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerStarted","Data":"81334220ecabd2882b05f8da017d52c375221e8279396f1d590162a7e640c0e8"}
Sep 05 02:29:59.081865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:59.081836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerStarted","Data":"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"}
Sep 05 02:29:59.081981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:29:59.081897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerStarted","Data":"98c428339094bfca495584f9c6567f0ee9d170277838858fbba32827fbd9a5f9"}
Sep 05 02:30:03.107111 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:03.107073    2560 generic.go:356] "Generic (PLEG): container finished" podID="d491b1eb-7607-4033-ac88-2138a2993925" containerID="6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6" exitCode=0
Sep 05 02:30:03.107448 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:03.107147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerDied","Data":"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"}
Sep 05 02:30:05.123254 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:05.123213    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/0.log"
Sep 05 02:30:05.123708 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:05.123258    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f289988-287c-485c-b78f-d51abc06bac1" containerID="d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b" exitCode=1
Sep 05 02:30:05.123708 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:05.123373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerDied","Data":"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"}
Sep 05 02:30:06.141279 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:06.141251    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/0.log"
Sep 05 02:30:06.141733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:06.141307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerStarted","Data":"19d1595500175d11bd2515c6d365ac223caadd71f459c40770eff48a8e2eb3fd"}
Sep 05 02:30:07.297675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:07.296873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:30:07.297675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:07.297139    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer" containerID="cri-o://19d1595500175d11bd2515c6d365ac223caadd71f459c40770eff48a8e2eb3fd" gracePeriod=30
Sep 05 02:30:12.178914 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:12.178872    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/1.log"
Sep 05 02:30:12.179398 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:12.179378    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/0.log"
Sep 05 02:30:12.179457 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:12.179419    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f289988-287c-485c-b78f-d51abc06bac1" containerID="19d1595500175d11bd2515c6d365ac223caadd71f459c40770eff48a8e2eb3fd" exitCode=1
Sep 05 02:30:12.179511 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:12.179494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerDied","Data":"19d1595500175d11bd2515c6d365ac223caadd71f459c40770eff48a8e2eb3fd"}
Sep 05 02:30:12.179738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:12.179559    2560 scope.go:117] "RemoveContainer" containerID="d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"
Sep 05 02:30:13.415152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.415122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/1.log"
Sep 05 02:30:13.417896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.417873    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:30:13.484252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484224    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location\") pod \"2f289988-287c-485c-b78f-d51abc06bac1\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") "
Sep 05 02:30:13.484413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") pod \"2f289988-287c-485c-b78f-d51abc06bac1\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") "
Sep 05 02:30:13.484413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484304    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config\") pod \"2f289988-287c-485c-b78f-d51abc06bac1\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") "
Sep 05 02:30:13.484547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484409    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert\") pod \"2f289988-287c-485c-b78f-d51abc06bac1\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") "
Sep 05 02:30:13.484547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484441    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b26tf\" (UniqueName: \"kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf\") pod \"2f289988-287c-485c-b78f-d51abc06bac1\" (UID: \"2f289988-287c-485c-b78f-d51abc06bac1\") "
Sep 05 02:30:13.484547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f289988-287c-485c-b78f-d51abc06bac1" (UID: "2f289988-287c-485c-b78f-d51abc06bac1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:30:13.484704 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f289988-287c-485c-b78f-d51abc06bac1" (UID: "2f289988-287c-485c-b78f-d51abc06bac1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:13.484812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484756    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f289988-287c-485c-b78f-d51abc06bac1" (UID: "2f289988-287c-485c-b78f-d51abc06bac1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:13.484812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484763    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f289988-287c-485c-b78f-d51abc06bac1-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:30:13.484812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.484804    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:30:13.486494 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.486469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f289988-287c-485c-b78f-d51abc06bac1" (UID: "2f289988-287c-485c-b78f-d51abc06bac1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:30:13.486769 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.486727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf" (OuterVolumeSpecName: "kube-api-access-b26tf") pod "2f289988-287c-485c-b78f-d51abc06bac1" (UID: "2f289988-287c-485c-b78f-d51abc06bac1"). InnerVolumeSpecName "kube-api-access-b26tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:30:13.586086 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.586050    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f289988-287c-485c-b78f-d51abc06bac1-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:30:13.586086 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.586084    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b26tf\" (UniqueName: \"kubernetes.io/projected/2f289988-287c-485c-b78f-d51abc06bac1-kube-api-access-b26tf\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:30:13.586357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:13.586094    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f289988-287c-485c-b78f-d51abc06bac1-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:30:14.192206 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.192176    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-827065-predictor-69c84ddc55-8wwcw_2f289988-287c-485c-b78f-d51abc06bac1/storage-initializer/1.log"
Sep 05 02:30:14.192658 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.192629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw" event={"ID":"2f289988-287c-485c-b78f-d51abc06bac1","Type":"ContainerDied","Data":"98c428339094bfca495584f9c6567f0ee9d170277838858fbba32827fbd9a5f9"}
Sep 05 02:30:14.192786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.192668    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"
Sep 05 02:30:14.230029 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.230000    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:30:14.233694 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.233668    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-827065-predictor-69c84ddc55-8wwcw"]
Sep 05 02:30:14.971455 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:14.971424    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f289988-287c-485c-b78f-d51abc06bac1" path="/var/lib/kubelet/pods/2f289988-287c-485c-b78f-d51abc06bac1/volumes"
Sep 05 02:30:16.405226 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:16.405055    2560 scope.go:117] "RemoveContainer" containerID="19d1595500175d11bd2515c6d365ac223caadd71f459c40770eff48a8e2eb3fd"
Sep 05 02:30:16.412655 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:16.412636    2560 scope.go:117] "RemoveContainer" containerID="d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"
Sep 05 02:30:16.412920 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:30:16.412902    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b\": container with ID starting with d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b not found: ID does not exist" containerID="d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"
Sep 05 02:30:16.412978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:16.412928    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b"} err="failed to get container status \"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b\": rpc error: code = NotFound desc = could not find container \"d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b\": container with ID starting with d9f4ad30f493e9dcb1c7cef396baa1647cf1baac683aee356fbc13ddbacfa60b not found: ID does not exist"
Sep 05 02:30:17.211644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:17.211596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerStarted","Data":"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"}
Sep 05 02:30:19.225304 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:19.225269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerStarted","Data":"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"}
Sep 05 02:30:21.239235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:21.239196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerStarted","Data":"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"}
Sep 05 02:30:21.239666 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:21.239324    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:30:21.259785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:21.259727    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podStartSLOduration=2.423381297 podStartE2EDuration="24.259694338s" totalImagesPullingTime="17.883195703s" totalInitContainerRuntime="3.953117338s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:21.258326279 +0000 UTC m=+764.897538110" watchObservedRunningTime="2026-09-05 02:30:21.259694338 +0000 UTC m=+764.898906138"
Sep 05 02:30:22.243775 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:22.243725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:30:22.244217 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:22.243808    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:30:22.245054 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:22.245017    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:22.245649 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:22.245626    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:23.249389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:23.249338    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:23.250979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:23.250948    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:23.256286 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:23.256262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:30:24.254082 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:24.254047    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:24.254433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:24.254306    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:34.254405 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:34.254364    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:34.254835 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:34.254796    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:44.254263 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:44.254224    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:44.254724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:44.254666    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:54.254436 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:54.254398    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:54.254898 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:30:54.254874    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:04.254525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:04.254439    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:31:04.254989 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:04.254886    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:14.254601 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:14.254561    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:31:14.255120 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:14.255096    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:24.254063 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:24.254021    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:31:24.254542 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:24.254520    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:34.254916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:34.254882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:31:34.255367 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:34.255028    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:31:42.463356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.463323    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:31:42.463787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.463573    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" containerID="cri-o://8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb" gracePeriod=30
Sep 05 02:31:42.463787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.463589    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" containerID="cri-o://5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448" gracePeriod=30
Sep 05 02:31:42.463787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.463629    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" containerID="cri-o://5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20" gracePeriod=30
Sep 05 02:31:42.586648 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.586617    2560 generic.go:356] "Generic (PLEG): container finished" podID="d491b1eb-7607-4033-ac88-2138a2993925" containerID="5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20" exitCode=2
Sep 05 02:31:42.586770 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.586662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerDied","Data":"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"}
Sep 05 02:31:42.602896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.602871    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:31:42.603196 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603184    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.603255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603197    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.603255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603211    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.603255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603216    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.603364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603271    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.603364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.603283    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f289988-287c-485c-b78f-d51abc06bac1" containerName="storage-initializer"
Sep 05 02:31:42.606236 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.606221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.608982 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.608963    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 05 02:31:42.608982 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.608978    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 05 02:31:42.617356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.617336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:31:42.788237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.788168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.788237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.788199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.788394 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.788293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.788394 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.788361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8f5h\" (UniqueName: \"kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.788462 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.788395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.889472 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.889608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8f5h\" (UniqueName: \"kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.889608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.889608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.889608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.890006 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.889983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.890163 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.890132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.890225 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.890197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.892063 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.892044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.899465 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.899431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8f5h\" (UniqueName: \"kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h\") pod \"isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:42.916299 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:42.916279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:43.194485 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:43.194448    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:31:43.197611 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:31:43.197585    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca4407a_62e0_4d52_9f0c_b13be5392b8d.slice/crio-a419cf7244d1f53166a12e36a33330ed972b6ddfeeb8add9aac10720c33b3197 WatchSource:0}: Error finding container a419cf7244d1f53166a12e36a33330ed972b6ddfeeb8add9aac10720c33b3197: Status 404 returned error can't find the container with id a419cf7244d1f53166a12e36a33330ed972b6ddfeeb8add9aac10720c33b3197
Sep 05 02:31:43.249022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:43.248994    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:31:43.593286 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:43.593191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerStarted","Data":"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"}
Sep 05 02:31:43.593630 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:43.593286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerStarted","Data":"a419cf7244d1f53166a12e36a33330ed972b6ddfeeb8add9aac10720c33b3197"}
Sep 05 02:31:44.254404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:44.254365    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:31:44.254761 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:44.254719    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:47.615249 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:47.615168    2560 generic.go:356] "Generic (PLEG): container finished" podID="d491b1eb-7607-4033-ac88-2138a2993925" containerID="8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb" exitCode=0
Sep 05 02:31:47.615679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:47.615246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerDied","Data":"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"}
Sep 05 02:31:47.616980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:47.616953    2560 generic.go:356] "Generic (PLEG): container finished" podID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerID="5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d" exitCode=0
Sep 05 02:31:47.617094 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:47.617017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerDied","Data":"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"}
Sep 05 02:31:48.249382 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.249336    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:31:48.628779 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.628684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerStarted","Data":"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"}
Sep 05 02:31:48.629138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.628791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerStarted","Data":"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"}
Sep 05 02:31:48.629138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.628818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerStarted","Data":"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"}
Sep 05 02:31:48.629219 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.629168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:48.629251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.629223    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:48.630513 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.630485    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:31:48.653577 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:48.653532    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podStartSLOduration=2.686182607 podStartE2EDuration="6.653520419s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.967337812s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:48.651276835 +0000 UTC m=+852.290488664" watchObservedRunningTime="2026-09-05 02:31:48.653520419 +0000 UTC m=+852.292732218"
Sep 05 02:31:49.634214 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:49.634170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:49.634670 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:49.634288    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:31:49.635316 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:49.635292    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:50.638464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:50.638420    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:31:50.638915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:50.638646    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:53.248997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:53.248956    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:31:53.249365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:53.249094    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:31:54.254731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:54.254696    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:31:54.255109 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:54.255019    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:55.649244 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:55.649210    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:31:55.649807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:55.649779    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:31:55.650199 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:55.650176    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:31:58.249058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:31:58.249012    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:32:03.249661 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:03.249607    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:32:04.254488 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:04.254449    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:32:04.254916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:04.254595    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:32:04.254916 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:04.254832    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:04.255001 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:04.254938    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:32:05.650107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:05.650066    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:05.650639 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:05.650576    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:08.249516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:08.249474    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:32:12.597627 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.597597    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:32:12.733130 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls\") pod \"d491b1eb-7607-4033-ac88-2138a2993925\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") "
Sep 05 02:32:12.733130 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert\") pod \"d491b1eb-7607-4033-ac88-2138a2993925\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") "
Sep 05 02:32:12.733348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config\") pod \"d491b1eb-7607-4033-ac88-2138a2993925\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") "
Sep 05 02:32:12.733348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngrsb\" (UniqueName: \"kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb\") pod \"d491b1eb-7607-4033-ac88-2138a2993925\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") "
Sep 05 02:32:12.733348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733234    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location\") pod \"d491b1eb-7607-4033-ac88-2138a2993925\" (UID: \"d491b1eb-7607-4033-ac88-2138a2993925\") "
Sep 05 02:32:12.733609 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733531    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d491b1eb-7607-4033-ac88-2138a2993925" (UID: "d491b1eb-7607-4033-ac88-2138a2993925"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:12.733736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733531    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d491b1eb-7607-4033-ac88-2138a2993925" (UID: "d491b1eb-7607-4033-ac88-2138a2993925"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:12.733736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.733691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d491b1eb-7607-4033-ac88-2138a2993925" (UID: "d491b1eb-7607-4033-ac88-2138a2993925"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:12.735333 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.735307    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb" (OuterVolumeSpecName: "kube-api-access-ngrsb") pod "d491b1eb-7607-4033-ac88-2138a2993925" (UID: "d491b1eb-7607-4033-ac88-2138a2993925"). InnerVolumeSpecName "kube-api-access-ngrsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:12.735584 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.735561    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d491b1eb-7607-4033-ac88-2138a2993925" (UID: "d491b1eb-7607-4033-ac88-2138a2993925"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:12.741369 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.741344    2560 generic.go:356] "Generic (PLEG): container finished" podID="d491b1eb-7607-4033-ac88-2138a2993925" containerID="5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448" exitCode=0
Sep 05 02:32:12.741470 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.741394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerDied","Data":"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"}
Sep 05 02:32:12.741470 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.741417    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"
Sep 05 02:32:12.741470 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.741453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb" event={"ID":"d491b1eb-7607-4033-ac88-2138a2993925","Type":"ContainerDied","Data":"81334220ecabd2882b05f8da017d52c375221e8279396f1d590162a7e640c0e8"}
Sep 05 02:32:12.741639 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.741492    2560 scope.go:117] "RemoveContainer" containerID="5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"
Sep 05 02:32:12.752486 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.752468    2560 scope.go:117] "RemoveContainer" containerID="5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"
Sep 05 02:32:12.759715 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.759697    2560 scope.go:117] "RemoveContainer" containerID="8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"
Sep 05 02:32:12.765232 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.765212    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:32:12.767624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.767602    2560 scope.go:117] "RemoveContainer" containerID="6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"
Sep 05 02:32:12.769812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.769790    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b79d4588c-22rzb"]
Sep 05 02:32:12.774655 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.774640    2560 scope.go:117] "RemoveContainer" containerID="5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"
Sep 05 02:32:12.775004 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:32:12.774983    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448\": container with ID starting with 5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448 not found: ID does not exist" containerID="5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"
Sep 05 02:32:12.775071 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775009    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448"} err="failed to get container status \"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448\": rpc error: code = NotFound desc = could not find container \"5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448\": container with ID starting with 5ea82f58c6bbd5935b7363437d3e07dc6c838ffbee656c0d2f3a602e8130b448 not found: ID does not exist"
Sep 05 02:32:12.775071 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775025    2560 scope.go:117] "RemoveContainer" containerID="5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"
Sep 05 02:32:12.775266 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:32:12.775252    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20\": container with ID starting with 5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20 not found: ID does not exist" containerID="5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"
Sep 05 02:32:12.775306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775270    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20"} err="failed to get container status \"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20\": rpc error: code = NotFound desc = could not find container \"5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20\": container with ID starting with 5b908770bc5e1bdcefe021903de4c829f796c68537ffc9016acd0d400fdbdd20 not found: ID does not exist"
Sep 05 02:32:12.775306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775281    2560 scope.go:117] "RemoveContainer" containerID="8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"
Sep 05 02:32:12.775484 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:32:12.775469    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb\": container with ID starting with 8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb not found: ID does not exist" containerID="8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"
Sep 05 02:32:12.775530 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775488    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb"} err="failed to get container status \"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb\": rpc error: code = NotFound desc = could not find container \"8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb\": container with ID starting with 8bf096b71433efa1341224f824ea3ed4b6a5af4a45c44efba611f6c7a977fdcb not found: ID does not exist"
Sep 05 02:32:12.775530 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775499    2560 scope.go:117] "RemoveContainer" containerID="6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"
Sep 05 02:32:12.775731 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:32:12.775716    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6\": container with ID starting with 6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6 not found: ID does not exist" containerID="6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"
Sep 05 02:32:12.775814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.775733    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6"} err="failed to get container status \"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6\": rpc error: code = NotFound desc = could not find container \"6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6\": container with ID starting with 6fe94acbc4088ee6431fb07653e1e826f03b116b0860352959724658510967a6 not found: ID does not exist"
Sep 05 02:32:12.834319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.834296    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d491b1eb-7607-4033-ac88-2138a2993925-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:32:12.834319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.834316    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:32:12.834441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.834325    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d491b1eb-7607-4033-ac88-2138a2993925-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:32:12.834441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.834334    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngrsb\" (UniqueName: \"kubernetes.io/projected/d491b1eb-7607-4033-ac88-2138a2993925-kube-api-access-ngrsb\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:32:12.834441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.834343    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d491b1eb-7607-4033-ac88-2138a2993925-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:32:12.971636 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:12.971610    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d491b1eb-7607-4033-ac88-2138a2993925" path="/var/lib/kubelet/pods/d491b1eb-7607-4033-ac88-2138a2993925/volumes"
Sep 05 02:32:15.650004 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:15.649965    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:15.650450 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:15.650425    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:25.649892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:25.649847    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:25.650273 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:25.650222    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:35.649896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:35.649799    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:35.650332 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:35.650309    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:36.798856 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:36.798820    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:32:36.801474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:36.801452    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:32:45.649731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:45.649690    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:45.651864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:45.650129    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:32:55.649877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:55.649828    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:32:55.650365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:32:55.650336    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:01.965925 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:01.965894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:01.966328 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:01.966168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:07.618923 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.618884    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:33:07.619440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.619187    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" containerID="cri-o://a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087" gracePeriod=30
Sep 05 02:33:07.619440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.619225    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" containerID="cri-o://5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230" gracePeriod=30
Sep 05 02:33:07.619440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.619269    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" containerID="cri-o://7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d" gracePeriod=30
Sep 05 02:33:07.990467 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.990433    2560 generic.go:356] "Generic (PLEG): container finished" podID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerID="7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d" exitCode=2
Sep 05 02:33:07.990644 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:07.990507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerDied","Data":"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"}
Sep 05 02:33:10.639070 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:10.639027    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:11.965816 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:11.965774    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:33:11.966196 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:11.966118    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:12.009567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:12.009540    2560 generic.go:356] "Generic (PLEG): container finished" podID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerID="a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087" exitCode=0
Sep 05 02:33:12.009674 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:12.009599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerDied","Data":"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"}
Sep 05 02:33:15.639073 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:15.639034    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:20.638651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:20.638609    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:20.639087 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:20.638785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:21.965764 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:21.965702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:33:21.966182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:21.966082    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:25.638975 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:25.638930    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:27.759014 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.758978    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759285    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759295    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759313    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="storage-initializer"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759318    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="storage-initializer"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759325    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759330    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759345    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container"
Sep 05 02:33:27.759360 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759350    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container"
Sep 05 02:33:27.759614 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759398    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kserve-container"
Sep 05 02:33:27.759614 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759408    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="kube-rbac-proxy"
Sep 05 02:33:27.759614 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.759414    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d491b1eb-7607-4033-ac88-2138a2993925" containerName="agent"
Sep 05 02:33:27.762868 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.762851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.765374 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.765349    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 05 02:33:27.765498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.765480    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 05 02:33:27.773523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.773500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:33:27.800414 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.800389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.800523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.800443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.800579 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.800548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcljr\" (UniqueName: \"kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.800619 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.800596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.800676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.800655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.901637 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.901614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.901734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.901648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.901734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.901677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.901734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.901728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcljr\" (UniqueName: \"kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.901983 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.901782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.902129 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.902107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.902348 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.902331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.902393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.902372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.904120 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.904105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:27.910079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:27.910054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcljr\" (UniqueName: \"kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr\") pod \"isvc-logger-predictor-7fd97f8756-z5cbw\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:28.073581 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:28.073522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:28.345253 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:28.345202    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:33:28.348051 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:33:28.348022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7048636_d90f_464b_b5ff_629148b4ac08.slice/crio-3ccf4232fb9957a2408651ef6575f62fcf3b342aca6680f352ca83fd6be3caac WatchSource:0}: Error finding container 3ccf4232fb9957a2408651ef6575f62fcf3b342aca6680f352ca83fd6be3caac: Status 404 returned error can't find the container with id 3ccf4232fb9957a2408651ef6575f62fcf3b342aca6680f352ca83fd6be3caac
Sep 05 02:33:29.079261 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:29.079224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerStarted","Data":"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"}
Sep 05 02:33:29.079607 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:29.079285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerStarted","Data":"3ccf4232fb9957a2408651ef6575f62fcf3b342aca6680f352ca83fd6be3caac"}
Sep 05 02:33:30.638802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:30.638734    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:31.966533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:31.966487    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Sep 05 02:33:31.966928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:31.966640    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:31.966928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:31.966874    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:31.967028 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:31.966986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:33.097879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:33.097848    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7048636-d90f-464b-b5ff-629148b4ac08" containerID="f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e" exitCode=0
Sep 05 02:33:33.098256 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:33.097919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerDied","Data":"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"}
Sep 05 02:33:34.105547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.105515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerStarted","Data":"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"}
Sep 05 02:33:34.105959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.105582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerStarted","Data":"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"}
Sep 05 02:33:34.105959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.105607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerStarted","Data":"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"}
Sep 05 02:33:34.105959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.105894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:34.106076 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.106045    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:34.107528 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.107499    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:33:34.127016 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:34.126974    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podStartSLOduration=3.344434792 podStartE2EDuration="7.126960156s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782525364s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:33:34.125257255 +0000 UTC m=+957.764469074" watchObservedRunningTime="2026-09-05 02:33:34.126960156 +0000 UTC m=+957.766171958"
Sep 05 02:33:35.109971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:35.109922    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:33:35.110405 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:35.109970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:35.111166 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:35.111130    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:35.639495 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:35.639450    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:33:36.115150 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:36.115044    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:33:36.115521 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:36.115465    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:37.764262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.764237    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:37.879380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879306    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config\") pod \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") "
Sep 05 02:33:37.879380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879353    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert\") pod \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") "
Sep 05 02:33:37.879593 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8f5h\" (UniqueName: \"kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h\") pod \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") "
Sep 05 02:33:37.879593 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879441    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location\") pod \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") "
Sep 05 02:33:37.879593 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls\") pod \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\" (UID: \"eca4407a-62e0-4d52-9f0c-b13be5392b8d\") "
Sep 05 02:33:37.879779 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eca4407a-62e0-4d52-9f0c-b13be5392b8d" (UID: "eca4407a-62e0-4d52-9f0c-b13be5392b8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:37.879841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879754    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eca4407a-62e0-4d52-9f0c-b13be5392b8d" (UID: "eca4407a-62e0-4d52-9f0c-b13be5392b8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:37.879841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eca4407a-62e0-4d52-9f0c-b13be5392b8d" (UID: "eca4407a-62e0-4d52-9f0c-b13be5392b8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:37.879952 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879844    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:33:37.879952 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879864    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca4407a-62e0-4d52-9f0c-b13be5392b8d-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:33:37.879952 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.879876    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:33:37.881529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.881493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eca4407a-62e0-4d52-9f0c-b13be5392b8d" (UID: "eca4407a-62e0-4d52-9f0c-b13be5392b8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:37.881529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.881498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h" (OuterVolumeSpecName: "kube-api-access-j8f5h") pod "eca4407a-62e0-4d52-9f0c-b13be5392b8d" (UID: "eca4407a-62e0-4d52-9f0c-b13be5392b8d"). InnerVolumeSpecName "kube-api-access-j8f5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:37.980538 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.980516    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8f5h\" (UniqueName: \"kubernetes.io/projected/eca4407a-62e0-4d52-9f0c-b13be5392b8d-kube-api-access-j8f5h\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:33:37.980538 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:37.980537    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca4407a-62e0-4d52-9f0c-b13be5392b8d-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:33:38.126368 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.126335    2560 generic.go:356] "Generic (PLEG): container finished" podID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerID="5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230" exitCode=0
Sep 05 02:33:38.126531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.126389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerDied","Data":"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"}
Sep 05 02:33:38.126531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.126397    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"
Sep 05 02:33:38.126531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.126460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b" event={"ID":"eca4407a-62e0-4d52-9f0c-b13be5392b8d","Type":"ContainerDied","Data":"a419cf7244d1f53166a12e36a33330ed972b6ddfeeb8add9aac10720c33b3197"}
Sep 05 02:33:38.126531 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.126505    2560 scope.go:117] "RemoveContainer" containerID="5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"
Sep 05 02:33:38.135009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.134991    2560 scope.go:117] "RemoveContainer" containerID="7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"
Sep 05 02:33:38.142528 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.142513    2560 scope.go:117] "RemoveContainer" containerID="a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"
Sep 05 02:33:38.149463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.149393    2560 scope.go:117] "RemoveContainer" containerID="5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"
Sep 05 02:33:38.151941 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.151918    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:33:38.155570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.155543    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b6465fdcd-nfk6b"]
Sep 05 02:33:38.157040 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157021    2560 scope.go:117] "RemoveContainer" containerID="5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"
Sep 05 02:33:38.157306 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:33:38.157291    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230\": container with ID starting with 5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230 not found: ID does not exist" containerID="5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"
Sep 05 02:33:38.157344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230"} err="failed to get container status \"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230\": rpc error: code = NotFound desc = could not find container \"5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230\": container with ID starting with 5869cc896ece96ea1c4c547cf6d4989dad6ce42889c7e72fad4eaf7f14f41230 not found: ID does not exist"
Sep 05 02:33:38.157344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157329    2560 scope.go:117] "RemoveContainer" containerID="7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"
Sep 05 02:33:38.157534 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:33:38.157519    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d\": container with ID starting with 7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d not found: ID does not exist" containerID="7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"
Sep 05 02:33:38.157577 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157539    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d"} err="failed to get container status \"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d\": rpc error: code = NotFound desc = could not find container \"7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d\": container with ID starting with 7b3dafce429d1f6bf38d3866247c86750b544f07817fbe3c80b75c32925f301d not found: ID does not exist"
Sep 05 02:33:38.157577 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157551    2560 scope.go:117] "RemoveContainer" containerID="a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"
Sep 05 02:33:38.157759 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:33:38.157732    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087\": container with ID starting with a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087 not found: ID does not exist" containerID="a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"
Sep 05 02:33:38.157799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157764    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087"} err="failed to get container status \"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087\": rpc error: code = NotFound desc = could not find container \"a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087\": container with ID starting with a5dd04cb763fd66401f05cdbcf1c0b52aa0932b339d022a211544edeaf7aa087 not found: ID does not exist"
Sep 05 02:33:38.157799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157776    2560 scope.go:117] "RemoveContainer" containerID="5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"
Sep 05 02:33:38.157968 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:33:38.157953    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d\": container with ID starting with 5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d not found: ID does not exist" containerID="5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"
Sep 05 02:33:38.158005 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.157973    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d"} err="failed to get container status \"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d\": rpc error: code = NotFound desc = could not find container \"5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d\": container with ID starting with 5820c25a35a2b9cb3e44fdf20e768e46d603f66a599eee3dcaba10332b922e8d not found: ID does not exist"
Sep 05 02:33:38.970637 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:38.970605    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" path="/var/lib/kubelet/pods/eca4407a-62e0-4d52-9f0c-b13be5392b8d/volumes"
Sep 05 02:33:41.119773 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:41.119719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:33:41.120237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:41.120213    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:33:41.120544 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:41.120511    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:33:51.120607 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:51.120567    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:33:51.121144 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:33:51.121119    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:01.120238 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:01.120146    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:34:01.120692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:01.120656    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:11.120523 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:11.120474    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:34:11.120996 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:11.120956    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:21.120246 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:21.120205    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:34:21.120657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:21.120575    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:31.120395 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:31.120360    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:34:31.120860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:31.120820    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:41.121005 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:41.120963    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:34:41.121536 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:41.121512    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:51.121547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:51.121510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:34:51.122067 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:34:51.121735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:02.940914 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:02.940880    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:35:02.941366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:02.941190    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" containerID="cri-o://b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91" gracePeriod=30
Sep 05 02:35:02.941366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:02.941268    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" containerID="cri-o://630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca" gracePeriod=30
Sep 05 02:35:02.941641 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:02.941295    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" containerID="cri-o://a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b" gracePeriod=30
Sep 05 02:35:03.500789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:03.500756    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7048636-d90f-464b-b5ff-629148b4ac08" containerID="a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b" exitCode=2
Sep 05 02:35:03.500945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:03.500817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerDied","Data":"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"}
Sep 05 02:35:06.115316 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:06.115277    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:07.522292 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:07.522245    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7048636-d90f-464b-b5ff-629148b4ac08" containerID="b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91" exitCode=0
Sep 05 02:35:07.522646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:07.522319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerDied","Data":"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"}
Sep 05 02:35:11.115229 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:11.115181    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:11.120623 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:11.120575    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:35:11.120929 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:11.120899    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:16.115664 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:16.115622    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:16.116129 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:16.115792    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:18.296682 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.296647    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.296979    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="storage-initializer"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.296992    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="storage-initializer"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.296999    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297005    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297012    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297017    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297034    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297039    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297089    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kube-rbac-proxy"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297096    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="kserve-container"
Sep 05 02:35:18.297131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.297106    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca4407a-62e0-4d52-9f0c-b13be5392b8d" containerName="agent"
Sep 05 02:35:18.300135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.300117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.302435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.302411    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 05 02:35:18.302435 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.302410    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 05 02:35:18.310257 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.310234    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:35:18.377509 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.377480    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2k5\" (UniqueName: \"kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.377625 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.377543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.377625 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.377597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.377733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.377622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.377733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.377677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.478602 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.478563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.478731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.478632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.478731 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.478683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2k5\" (UniqueName: \"kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.478892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.478774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.478892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.478819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.479146 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.479125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.479319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.479298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.479361 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.479320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.481128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.481107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.485990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.485962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2k5\" (UniqueName: \"kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.610851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.610796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:18.886212 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:18.886188    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:35:18.888230 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:35:18.888201    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc7070e_f792_489c_95b1_ecb25819ce67.slice/crio-e03c9d422a60926c3d9f5b51c648a9a372423ba9e22fedd8d624a04f44bb2771 WatchSource:0}: Error finding container e03c9d422a60926c3d9f5b51c648a9a372423ba9e22fedd8d624a04f44bb2771: Status 404 returned error can't find the container with id e03c9d422a60926c3d9f5b51c648a9a372423ba9e22fedd8d624a04f44bb2771
Sep 05 02:35:19.575990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:19.575949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerStarted","Data":"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"}
Sep 05 02:35:19.576426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:19.576065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerStarted","Data":"e03c9d422a60926c3d9f5b51c648a9a372423ba9e22fedd8d624a04f44bb2771"}
Sep 05 02:35:21.115766 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:21.115705    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:21.121090 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:21.121053    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:35:21.121478 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:21.121448    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:23.596687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:23.596652    2560 generic.go:356] "Generic (PLEG): container finished" podID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerID="c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af" exitCode=0
Sep 05 02:35:23.597114 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:23.596730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerDied","Data":"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"}
Sep 05 02:35:23.598096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:23.598082    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:35:26.115677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:26.115445    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:31.115487 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:31.115334    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:35:31.120816 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:31.120769    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:35:31.120963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:31.120926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:31.121098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:31.121075    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:31.121250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:31.121192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:34.616842 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.616809    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:34.671364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.671328    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7048636-d90f-464b-b5ff-629148b4ac08" containerID="630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca" exitCode=0
Sep 05 02:35:34.671525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.671380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerDied","Data":"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"}
Sep 05 02:35:34.671525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.671391    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"
Sep 05 02:35:34.671525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.671455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw" event={"ID":"a7048636-d90f-464b-b5ff-629148b4ac08","Type":"ContainerDied","Data":"3ccf4232fb9957a2408651ef6575f62fcf3b342aca6680f352ca83fd6be3caac"}
Sep 05 02:35:34.671525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.671494    2560 scope.go:117] "RemoveContainer" containerID="630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"
Sep 05 02:35:34.680968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.680943    2560 scope.go:117] "RemoveContainer" containerID="a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"
Sep 05 02:35:34.689118 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.689104    2560 scope.go:117] "RemoveContainer" containerID="b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"
Sep 05 02:35:34.698237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.698219    2560 scope.go:117] "RemoveContainer" containerID="f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"
Sep 05 02:35:34.706903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.706887    2560 scope.go:117] "RemoveContainer" containerID="630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"
Sep 05 02:35:34.707201 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:35:34.707177    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca\": container with ID starting with 630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca not found: ID does not exist" containerID="630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"
Sep 05 02:35:34.707302 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.707214    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca"} err="failed to get container status \"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca\": rpc error: code = NotFound desc = could not find container \"630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca\": container with ID starting with 630e165182cffcad2994f0043f4bd001942daca2153d3bc81e91d5de8d757bca not found: ID does not exist"
Sep 05 02:35:34.707302 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.707238    2560 scope.go:117] "RemoveContainer" containerID="a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"
Sep 05 02:35:34.707559 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:35:34.707541    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b\": container with ID starting with a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b not found: ID does not exist" containerID="a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"
Sep 05 02:35:34.707667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.707562    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b"} err="failed to get container status \"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b\": rpc error: code = NotFound desc = could not find container \"a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b\": container with ID starting with a4c161b975e664418d3724c68b24e1695471861df18d5012694f487a9b69995b not found: ID does not exist"
Sep 05 02:35:34.707667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.707578    2560 scope.go:117] "RemoveContainer" containerID="b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"
Sep 05 02:35:34.708128 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:35:34.708107    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91\": container with ID starting with b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91 not found: ID does not exist" containerID="b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"
Sep 05 02:35:34.708203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.708135    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91"} err="failed to get container status \"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91\": rpc error: code = NotFound desc = could not find container \"b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91\": container with ID starting with b2ee59a488b6a2a6b371352fcd4879e4408de9dc137c5915940e4682eccbfe91 not found: ID does not exist"
Sep 05 02:35:34.708203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.708154    2560 scope.go:117] "RemoveContainer" containerID="f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"
Sep 05 02:35:34.708461 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:35:34.708428    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e\": container with ID starting with f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e not found: ID does not exist" containerID="f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"
Sep 05 02:35:34.708527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.708468    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e"} err="failed to get container status \"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e\": rpc error: code = NotFound desc = could not find container \"f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e\": container with ID starting with f87eddd87aac0ed3868e48e74938a303e29ec55501446523627c56d54c12092e not found: ID does not exist"
Sep 05 02:35:34.720346 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcljr\" (UniqueName: \"kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr\") pod \"a7048636-d90f-464b-b5ff-629148b4ac08\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") "
Sep 05 02:35:34.720456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location\") pod \"a7048636-d90f-464b-b5ff-629148b4ac08\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") "
Sep 05 02:35:34.720456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls\") pod \"a7048636-d90f-464b-b5ff-629148b4ac08\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") "
Sep 05 02:35:34.720456 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert\") pod \"a7048636-d90f-464b-b5ff-629148b4ac08\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") "
Sep 05 02:35:34.720628 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config\") pod \"a7048636-d90f-464b-b5ff-629148b4ac08\" (UID: \"a7048636-d90f-464b-b5ff-629148b4ac08\") "
Sep 05 02:35:34.720685 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7048636-d90f-464b-b5ff-629148b4ac08" (UID: "a7048636-d90f-464b-b5ff-629148b4ac08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:34.720833 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720817    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7048636-d90f-464b-b5ff-629148b4ac08-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:35:34.720984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.720959    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7048636-d90f-464b-b5ff-629148b4ac08" (UID: "a7048636-d90f-464b-b5ff-629148b4ac08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:34.721064 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.721008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7048636-d90f-464b-b5ff-629148b4ac08" (UID: "a7048636-d90f-464b-b5ff-629148b4ac08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:34.722521 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.722496    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7048636-d90f-464b-b5ff-629148b4ac08" (UID: "a7048636-d90f-464b-b5ff-629148b4ac08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:34.722737 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.722719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr" (OuterVolumeSpecName: "kube-api-access-wcljr") pod "a7048636-d90f-464b-b5ff-629148b4ac08" (UID: "a7048636-d90f-464b-b5ff-629148b4ac08"). InnerVolumeSpecName "kube-api-access-wcljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:34.821396 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.821367    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcljr\" (UniqueName: \"kubernetes.io/projected/a7048636-d90f-464b-b5ff-629148b4ac08-kube-api-access-wcljr\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:35:34.821529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.821398    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7048636-d90f-464b-b5ff-629148b4ac08-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:35:34.821529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.821414    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:35:34.821529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.821430    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7048636-d90f-464b-b5ff-629148b4ac08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:35:34.998103 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:34.998074    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:35:35.001494 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:35.001472    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fd97f8756-z5cbw"]
Sep 05 02:35:36.971173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:36.971138    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" path="/var/lib/kubelet/pods/a7048636-d90f-464b-b5ff-629148b4ac08/volumes"
Sep 05 02:35:44.723684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:44.723644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerStarted","Data":"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"}
Sep 05 02:35:44.724105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:44.723729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerStarted","Data":"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"}
Sep 05 02:35:44.724105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:44.723950    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:44.724105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:44.724011    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:35:44.742268 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:44.742226    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podStartSLOduration=1.932639926 podStartE2EDuration="26.742213827s" totalImagesPullingTime="20.769817114s" totalInitContainerRuntime="4.039756787s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:44.740595113 +0000 UTC m=+1088.379806941" watchObservedRunningTime="2026-09-05 02:35:44.742213827 +0000 UTC m=+1088.381425626"
Sep 05 02:35:50.732521 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:35:50.732492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:36:20.733877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:36:20.733834    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:36:30.733606 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:36:30.733550    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:36:40.733402 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:36:40.733360    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:36:50.733940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:36:50.733901    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:37:00.836776 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:00.836660    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:37:08.439200 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:08.439167    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:37:08.439653 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:08.439504    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" containerID="cri-o://4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a" gracePeriod=30
Sep 05 02:37:08.439653 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:08.439615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kube-rbac-proxy" containerID="cri-o://29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7" gracePeriod=30
Sep 05 02:37:09.080942 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:09.080906    2560 generic.go:356] "Generic (PLEG): container finished" podID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerID="29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7" exitCode=2
Sep 05 02:37:09.081107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:09.080949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerDied","Data":"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"}
Sep 05 02:37:10.728049 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:10.727986    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 05 02:37:10.733395 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:10.733359    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:37:12.475585 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.475564    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:37:12.574473 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r2k5\" (UniqueName: \"kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5\") pod \"2cc7070e-f792-489c-95b1-ecb25819ce67\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") "
Sep 05 02:37:12.574591 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert\") pod \"2cc7070e-f792-489c-95b1-ecb25819ce67\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") "
Sep 05 02:37:12.574591 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config\") pod \"2cc7070e-f792-489c-95b1-ecb25819ce67\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") "
Sep 05 02:37:12.574591 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location\") pod \"2cc7070e-f792-489c-95b1-ecb25819ce67\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") "
Sep 05 02:37:12.574728 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574592    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls\") pod \"2cc7070e-f792-489c-95b1-ecb25819ce67\" (UID: \"2cc7070e-f792-489c-95b1-ecb25819ce67\") "
Sep 05 02:37:12.574918 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574879    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2cc7070e-f792-489c-95b1-ecb25819ce67" (UID: "2cc7070e-f792-489c-95b1-ecb25819ce67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:12.574918 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2cc7070e-f792-489c-95b1-ecb25819ce67" (UID: "2cc7070e-f792-489c-95b1-ecb25819ce67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:12.574918 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.574908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2cc7070e-f792-489c-95b1-ecb25819ce67" (UID: "2cc7070e-f792-489c-95b1-ecb25819ce67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:12.576573 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.576550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5" (OuterVolumeSpecName: "kube-api-access-5r2k5") pod "2cc7070e-f792-489c-95b1-ecb25819ce67" (UID: "2cc7070e-f792-489c-95b1-ecb25819ce67"). InnerVolumeSpecName "kube-api-access-5r2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:37:12.576654 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.576595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cc7070e-f792-489c-95b1-ecb25819ce67" (UID: "2cc7070e-f792-489c-95b1-ecb25819ce67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:37:12.675504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.675474    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2k5\" (UniqueName: \"kubernetes.io/projected/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-api-access-5r2k5\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:37:12.675504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.675503    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:37:12.675504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.675513    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cc7070e-f792-489c-95b1-ecb25819ce67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:37:12.675680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.675521    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cc7070e-f792-489c-95b1-ecb25819ce67-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:37:12.675680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:12.675530    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cc7070e-f792-489c-95b1-ecb25819ce67-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:37:13.099208 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.099175    2560 generic.go:356] "Generic (PLEG): container finished" podID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerID="4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a" exitCode=0
Sep 05 02:37:13.099389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.099228    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"
Sep 05 02:37:13.099389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.099250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerDied","Data":"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"}
Sep 05 02:37:13.099389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.099324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p" event={"ID":"2cc7070e-f792-489c-95b1-ecb25819ce67","Type":"ContainerDied","Data":"e03c9d422a60926c3d9f5b51c648a9a372423ba9e22fedd8d624a04f44bb2771"}
Sep 05 02:37:13.099559 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.099364    2560 scope.go:117] "RemoveContainer" containerID="29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"
Sep 05 02:37:13.107608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.107587    2560 scope.go:117] "RemoveContainer" containerID="4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"
Sep 05 02:37:13.114883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.114857    2560 scope.go:117] "RemoveContainer" containerID="c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"
Sep 05 02:37:13.118845 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.118823    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:37:13.122664 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.122636    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szr2p"]
Sep 05 02:37:13.122715 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.122694    2560 scope.go:117] "RemoveContainer" containerID="29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"
Sep 05 02:37:13.122954 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:37:13.122938    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7\": container with ID starting with 29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7 not found: ID does not exist" containerID="29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"
Sep 05 02:37:13.123003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.122962    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7"} err="failed to get container status \"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7\": rpc error: code = NotFound desc = could not find container \"29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7\": container with ID starting with 29e10cc132bbca044e18679a3622d5986c4247ed99179dca1c034618c852eaa7 not found: ID does not exist"
Sep 05 02:37:13.123003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.122977    2560 scope.go:117] "RemoveContainer" containerID="4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"
Sep 05 02:37:13.123192 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:37:13.123179    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a\": container with ID starting with 4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a not found: ID does not exist" containerID="4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"
Sep 05 02:37:13.123238 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.123196    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a"} err="failed to get container status \"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a\": rpc error: code = NotFound desc = could not find container \"4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a\": container with ID starting with 4c220bc9c0d064a00d62f3f4d215c012402fd819788185723168ad0f6793b47a not found: ID does not exist"
Sep 05 02:37:13.123238 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.123209    2560 scope.go:117] "RemoveContainer" containerID="c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"
Sep 05 02:37:13.123417 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:37:13.123404    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af\": container with ID starting with c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af not found: ID does not exist" containerID="c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"
Sep 05 02:37:13.123464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.123420    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af"} err="failed to get container status \"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af\": rpc error: code = NotFound desc = could not find container \"c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af\": container with ID starting with c38d70e4b7d982b596b5182d3d629f286dce275fc61e6ecc82d111fe3de665af not found: ID does not exist"
Sep 05 02:37:13.608418 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608719    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="storage-initializer"
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608730    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="storage-initializer"
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608754    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy"
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608763    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy"
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608777    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container"
Sep 05 02:37:13.608785 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608784    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608798    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="storage-initializer"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608803    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="storage-initializer"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608812    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608817    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608824    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kube-rbac-proxy"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608828    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kube-rbac-proxy"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608837    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608842    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608889    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="agent"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608897    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kube-rbac-proxy"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608904    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" containerName="kserve-container"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608911    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kserve-container"
Sep 05 02:37:13.608979 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.608917    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7048636-d90f-464b-b5ff-629148b4ac08" containerName="kube-rbac-proxy"
Sep 05 02:37:13.611789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.611772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.614344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.614316    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:37:13.614526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.614510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 05 02:37:13.614606 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.614548    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:37:13.614606 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.614554    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 05 02:37:13.615534 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.615515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:37:13.615615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.615539    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:37:13.615615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.615561    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:37:13.622802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.622780    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:37:13.683580 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.683557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjrn\" (UniqueName: \"kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.683686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.683608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.683686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.683670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.683786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.683694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.683786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.683738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784415 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjrn\" (UniqueName: \"kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784516 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.784966 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.784943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.785235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.785215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.785290 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.785215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.786815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.786788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.793029 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.793008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjrn\" (UniqueName: \"kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:13.921539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:13.921519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:14.195141 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:14.195113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:37:14.196593 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:37:14.196564    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965d709b_4e7f_4132_9a70_952572d96058.slice/crio-788e3d23312d2dc619622c383ceda13c31a5a13faf1a8f37f8b8d0dfabee1dd0 WatchSource:0}: Error finding container 788e3d23312d2dc619622c383ceda13c31a5a13faf1a8f37f8b8d0dfabee1dd0: Status 404 returned error can't find the container with id 788e3d23312d2dc619622c383ceda13c31a5a13faf1a8f37f8b8d0dfabee1dd0
Sep 05 02:37:14.970970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:14.970927    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc7070e-f792-489c-95b1-ecb25819ce67" path="/var/lib/kubelet/pods/2cc7070e-f792-489c-95b1-ecb25819ce67/volumes"
Sep 05 02:37:15.112532 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:15.112495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerStarted","Data":"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"}
Sep 05 02:37:15.112662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:15.112557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerStarted","Data":"788e3d23312d2dc619622c383ceda13c31a5a13faf1a8f37f8b8d0dfabee1dd0"}
Sep 05 02:37:18.128250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:18.128216    2560 generic.go:356] "Generic (PLEG): container finished" podID="965d709b-4e7f-4132-9a70-952572d96058" containerID="9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5" exitCode=0
Sep 05 02:37:18.128561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:18.128263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerDied","Data":"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"}
Sep 05 02:37:19.135335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:19.135294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerStarted","Data":"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"}
Sep 05 02:37:19.135684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:19.135375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerStarted","Data":"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"}
Sep 05 02:37:19.135684 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:19.135595    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:19.157480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:19.157435    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podStartSLOduration=2.416361641 podStartE2EDuration="6.157422728s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741061087s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:37:19.155056816 +0000 UTC m=+1182.794268617" watchObservedRunningTime="2026-09-05 02:37:19.157422728 +0000 UTC m=+1182.796634530"
Sep 05 02:37:20.140542 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:20.140509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:26.150121 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:26.150092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:37:36.826920 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:36.826888    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:37:36.830181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:36.830159    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:37:56.151163 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:37:56.151123    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:38:06.151061 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:06.151021    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:38:16.150738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:16.150692    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:38:26.150766 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:26.150712    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:38:36.237152 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:36.237069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:38:43.679837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:43.679807    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:38:43.680263 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:43.680022    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" containerID="cri-o://bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9" gracePeriod=30
Sep 05 02:38:43.680263 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:43.680068    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kube-rbac-proxy" containerID="cri-o://dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2" gracePeriod=30
Sep 05 02:38:44.488896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:44.488846    2560 generic.go:356] "Generic (PLEG): container finished" podID="965d709b-4e7f-4132-9a70-952572d96058" containerID="dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2" exitCode=2
Sep 05 02:38:44.489103 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:44.488917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerDied","Data":"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"}
Sep 05 02:38:46.145525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:46.145483    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 05 02:38:46.150891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:46.150857    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:38:47.919212 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:47.919189    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:38:48.007981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.007922    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls\") pod \"965d709b-4e7f-4132-9a70-952572d96058\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") "
Sep 05 02:38:48.007981 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.007977    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location\") pod \"965d709b-4e7f-4132-9a70-952572d96058\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") "
Sep 05 02:38:48.008140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config\") pod \"965d709b-4e7f-4132-9a70-952572d96058\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") "
Sep 05 02:38:48.008140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert\") pod \"965d709b-4e7f-4132-9a70-952572d96058\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") "
Sep 05 02:38:48.008224 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wjrn\" (UniqueName: \"kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn\") pod \"965d709b-4e7f-4132-9a70-952572d96058\" (UID: \"965d709b-4e7f-4132-9a70-952572d96058\") "
Sep 05 02:38:48.008378 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "965d709b-4e7f-4132-9a70-952572d96058" (UID: "965d709b-4e7f-4132-9a70-952572d96058"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:38:48.008478 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008379    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "965d709b-4e7f-4132-9a70-952572d96058" (UID: "965d709b-4e7f-4132-9a70-952572d96058"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:48.008478 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008386    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "965d709b-4e7f-4132-9a70-952572d96058" (UID: "965d709b-4e7f-4132-9a70-952572d96058"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:48.008576 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008540    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/965d709b-4e7f-4132-9a70-952572d96058-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.008576 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.008565    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.010080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.010059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "965d709b-4e7f-4132-9a70-952572d96058" (UID: "965d709b-4e7f-4132-9a70-952572d96058"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:38:48.010281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.010263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn" (OuterVolumeSpecName: "kube-api-access-2wjrn") pod "965d709b-4e7f-4132-9a70-952572d96058" (UID: "965d709b-4e7f-4132-9a70-952572d96058"). InnerVolumeSpecName "kube-api-access-2wjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:38:48.109033 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.109001    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/965d709b-4e7f-4132-9a70-952572d96058-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.109033 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.109029    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjrn\" (UniqueName: \"kubernetes.io/projected/965d709b-4e7f-4132-9a70-952572d96058-kube-api-access-2wjrn\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.109177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.109043    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/965d709b-4e7f-4132-9a70-952572d96058-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.506794 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.506761    2560 generic.go:356] "Generic (PLEG): container finished" podID="965d709b-4e7f-4132-9a70-952572d96058" containerID="bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9" exitCode=0
Sep 05 02:38:48.506946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.506817    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"
Sep 05 02:38:48.506946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.506842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerDied","Data":"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"}
Sep 05 02:38:48.506946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.506901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct" event={"ID":"965d709b-4e7f-4132-9a70-952572d96058","Type":"ContainerDied","Data":"788e3d23312d2dc619622c383ceda13c31a5a13faf1a8f37f8b8d0dfabee1dd0"}
Sep 05 02:38:48.506946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.506922    2560 scope.go:117] "RemoveContainer" containerID="dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"
Sep 05 02:38:48.515620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.515598    2560 scope.go:117] "RemoveContainer" containerID="bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"
Sep 05 02:38:48.522733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.522718    2560 scope.go:117] "RemoveContainer" containerID="9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"
Sep 05 02:38:48.529295 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.529275    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:38:48.530420 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.530402    2560 scope.go:117] "RemoveContainer" containerID="dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"
Sep 05 02:38:48.530680 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:38:48.530661    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2\": container with ID starting with dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2 not found: ID does not exist" containerID="dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"
Sep 05 02:38:48.530736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.530687    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2"} err="failed to get container status \"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2\": rpc error: code = NotFound desc = could not find container \"dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2\": container with ID starting with dfe2fc756eccf2e6f938e32e049cb8977793eac9afdc8f221c1ae3ca487a08f2 not found: ID does not exist"
Sep 05 02:38:48.530736 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.530704    2560 scope.go:117] "RemoveContainer" containerID="bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"
Sep 05 02:38:48.530978 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:38:48.530957    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9\": container with ID starting with bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9 not found: ID does not exist" containerID="bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"
Sep 05 02:38:48.531099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.530987    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9"} err="failed to get container status \"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9\": rpc error: code = NotFound desc = could not find container \"bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9\": container with ID starting with bbe3da22a954ec78e8d8aa832374321104e4b2584b88cd0624e80c7d742929e9 not found: ID does not exist"
Sep 05 02:38:48.531099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.531009    2560 scope.go:117] "RemoveContainer" containerID="9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"
Sep 05 02:38:48.531354 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:38:48.531331    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5\": container with ID starting with 9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5 not found: ID does not exist" containerID="9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"
Sep 05 02:38:48.531500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.531360    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5"} err="failed to get container status \"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5\": rpc error: code = NotFound desc = could not find container \"9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5\": container with ID starting with 9a17361853a83f8aee1848369505c05092992c476c249793e1729fab4f6653e5 not found: ID does not exist"
Sep 05 02:38:48.533311 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.533292    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2phct"]
Sep 05 02:38:48.857201 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:38:48.857561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857548    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="storage-initializer"
Sep 05 02:38:48.857603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857565    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="storage-initializer"
Sep 05 02:38:48.857603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857581    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container"
Sep 05 02:38:48.857603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857589    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container"
Sep 05 02:38:48.857696 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857611    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kube-rbac-proxy"
Sep 05 02:38:48.857696 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857620    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kube-rbac-proxy"
Sep 05 02:38:48.857776 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857720    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kserve-container"
Sep 05 02:38:48.857776 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.857733    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="965d709b-4e7f-4132-9a70-952572d96058" containerName="kube-rbac-proxy"
Sep 05 02:38:48.861057 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.861041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.863657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.863572    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 05 02:38:48.863657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.863571    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:38:48.863860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.863670    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:38:48.863860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.863677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:38:48.863860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.863678    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:38:48.864648 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.864628    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 05 02:38:48.864694 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.864668    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:38:48.869135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.869112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:38:48.915913 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.915885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.916044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.915918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.916044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.915966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.916044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.916033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69nn\" (UniqueName: \"kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.916187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.916074    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:48.970509 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:48.970484    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965d709b-4e7f-4132-9a70-952572d96058" path="/var/lib/kubelet/pods/965d709b-4e7f-4132-9a70-952572d96058/volumes"
Sep 05 02:38:49.016588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.016566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.016698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.016610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69nn\" (UniqueName: \"kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.016698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.016636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.016698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.016662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.016698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.016678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.017099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.017078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.017308 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.017290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.017357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.017312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.019066 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.019046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.024890 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.024871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69nn\" (UniqueName: \"kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.172109 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.172068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:49.446692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.446556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:38:49.449386 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:38:49.449355    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c599b6_7a46_4258_937f_b98239b35b05.slice/crio-878c5890ac5b62660719e4ccee61054ab939f861f1d85c51025b516eeec66ed7 WatchSource:0}: Error finding container 878c5890ac5b62660719e4ccee61054ab939f861f1d85c51025b516eeec66ed7: Status 404 returned error can't find the container with id 878c5890ac5b62660719e4ccee61054ab939f861f1d85c51025b516eeec66ed7
Sep 05 02:38:49.514211 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.514185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerStarted","Data":"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"}
Sep 05 02:38:49.514462 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:49.514246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerStarted","Data":"878c5890ac5b62660719e4ccee61054ab939f861f1d85c51025b516eeec66ed7"}
Sep 05 02:38:53.534108 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:53.534080    2560 generic.go:356] "Generic (PLEG): container finished" podID="73c599b6-7a46-4258-937f-b98239b35b05" containerID="1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b" exitCode=0
Sep 05 02:38:53.534420 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:53.534149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerDied","Data":"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"}
Sep 05 02:38:54.541484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:54.541448    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerStarted","Data":"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"}
Sep 05 02:38:54.541970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:54.541521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerStarted","Data":"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"}
Sep 05 02:38:54.541970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:54.541772    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:38:54.562327 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:54.562281    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podStartSLOduration=2.544239953 podStartE2EDuration="6.562266601s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.018026648s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:54.560706627 +0000 UTC m=+1278.199918448" watchObservedRunningTime="2026-09-05 02:38:54.562266601 +0000 UTC m=+1278.201478400"
Sep 05 02:38:55.546304 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:38:55.546269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:39:01.556828 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:39:01.556735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:39:31.558458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:39:31.558413    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:39:41.557673 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:39:41.557627    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:39:51.558257 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:39:51.558218    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:01.558074 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:01.557993    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:09.037013 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:09.036969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:40:18.932228 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:18.932194    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:40:18.932803 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:18.932484    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" containerID="cri-o://a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20" gracePeriod=30
Sep 05 02:40:18.932803 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:18.932511    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kube-rbac-proxy" containerID="cri-o://553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa" gracePeriod=30
Sep 05 02:40:18.966121 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:18.966087    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:19.919112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:19.919077    2560 generic.go:356] "Generic (PLEG): container finished" podID="73c599b6-7a46-4258-937f-b98239b35b05" containerID="553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa" exitCode=2
Sep 05 02:40:19.919274 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:19.919128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerDied","Data":"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"}
Sep 05 02:40:21.550870 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:21.550820    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 05 02:40:23.082012 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.081989    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:40:23.245042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.244964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69nn\" (UniqueName: \"kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn\") pod \"73c599b6-7a46-4258-937f-b98239b35b05\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") "
Sep 05 02:40:23.245042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert\") pod \"73c599b6-7a46-4258-937f-b98239b35b05\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") "
Sep 05 02:40:23.245254 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245059    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location\") pod \"73c599b6-7a46-4258-937f-b98239b35b05\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") "
Sep 05 02:40:23.245254 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config\") pod \"73c599b6-7a46-4258-937f-b98239b35b05\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") "
Sep 05 02:40:23.245254 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245112    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls\") pod \"73c599b6-7a46-4258-937f-b98239b35b05\" (UID: \"73c599b6-7a46-4258-937f-b98239b35b05\") "
Sep 05 02:40:23.245466 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73c599b6-7a46-4258-937f-b98239b35b05" (UID: "73c599b6-7a46-4258-937f-b98239b35b05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:23.245466 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73c599b6-7a46-4258-937f-b98239b35b05" (UID: "73c599b6-7a46-4258-937f-b98239b35b05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:23.245540 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.245444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73c599b6-7a46-4258-937f-b98239b35b05" (UID: "73c599b6-7a46-4258-937f-b98239b35b05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:23.247282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.247258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73c599b6-7a46-4258-937f-b98239b35b05" (UID: "73c599b6-7a46-4258-937f-b98239b35b05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:23.247282 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.247270    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn" (OuterVolumeSpecName: "kube-api-access-k69nn") pod "73c599b6-7a46-4258-937f-b98239b35b05" (UID: "73c599b6-7a46-4258-937f-b98239b35b05"). InnerVolumeSpecName "kube-api-access-k69nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:23.345883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.345856    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c599b6-7a46-4258-937f-b98239b35b05-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:23.345883 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.345880    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69nn\" (UniqueName: \"kubernetes.io/projected/73c599b6-7a46-4258-937f-b98239b35b05-kube-api-access-k69nn\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:23.346022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.345889    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:23.346022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.345900    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73c599b6-7a46-4258-937f-b98239b35b05-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:23.346022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.345909    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c599b6-7a46-4258-937f-b98239b35b05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:23.945000 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.944969    2560 generic.go:356] "Generic (PLEG): container finished" podID="73c599b6-7a46-4258-937f-b98239b35b05" containerID="a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20" exitCode=0
Sep 05 02:40:23.945158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.945028    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"
Sep 05 02:40:23.945158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.945037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerDied","Data":"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"}
Sep 05 02:40:23.945158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.945084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx" event={"ID":"73c599b6-7a46-4258-937f-b98239b35b05","Type":"ContainerDied","Data":"878c5890ac5b62660719e4ccee61054ab939f861f1d85c51025b516eeec66ed7"}
Sep 05 02:40:23.945158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.945110    2560 scope.go:117] "RemoveContainer" containerID="553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"
Sep 05 02:40:23.954222 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.954205    2560 scope.go:117] "RemoveContainer" containerID="a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"
Sep 05 02:40:23.962373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.962232    2560 scope.go:117] "RemoveContainer" containerID="1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"
Sep 05 02:40:23.970403 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.970378    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:40:23.970641 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.970621    2560 scope.go:117] "RemoveContainer" containerID="553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"
Sep 05 02:40:23.971136 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:40:23.971105    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa\": container with ID starting with 553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa not found: ID does not exist" containerID="553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"
Sep 05 02:40:23.971240 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.971144    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa"} err="failed to get container status \"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa\": rpc error: code = NotFound desc = could not find container \"553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa\": container with ID starting with 553afcedaa161c050b227ff35adbfa8b9ae240d21e8dd6b7d80a15744f07d9aa not found: ID does not exist"
Sep 05 02:40:23.971240 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.971169    2560 scope.go:117] "RemoveContainer" containerID="a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"
Sep 05 02:40:23.971445 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:40:23.971427    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20\": container with ID starting with a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20 not found: ID does not exist" containerID="a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"
Sep 05 02:40:23.971500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.971449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20"} err="failed to get container status \"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20\": rpc error: code = NotFound desc = could not find container \"a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20\": container with ID starting with a63568a2d52109518c76d45bd64e52909ae661ce839ec68b10f23ac6ec70ef20 not found: ID does not exist"
Sep 05 02:40:23.971500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.971465    2560 scope.go:117] "RemoveContainer" containerID="1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"
Sep 05 02:40:23.971699 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:40:23.971680    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b\": container with ID starting with 1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b not found: ID does not exist" containerID="1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"
Sep 05 02:40:23.971763 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.971703    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b"} err="failed to get container status \"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b\": rpc error: code = NotFound desc = could not find container \"1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b\": container with ID starting with 1cd206e22087080abb9c0e951a3a5dbb188e61e2e7166712557929d5216c492b not found: ID does not exist"
Sep 05 02:40:23.972507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:23.972487    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2z6vx"]
Sep 05 02:40:24.971000 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:24.970962    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c599b6-7a46-4258-937f-b98239b35b05" path="/var/lib/kubelet/pods/73c599b6-7a46-4258-937f-b98239b35b05/volumes"
Sep 05 02:40:41.647535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647501    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-9c5f057c"]
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647829    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="storage-initializer"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647841    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="storage-initializer"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647860    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647866    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647874    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kube-rbac-proxy"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647881    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kube-rbac-proxy"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647927    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kserve-container"
Sep 05 02:40:41.647940 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.647936    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c599b6-7a46-4258-937f-b98239b35b05" containerName="kube-rbac-proxy"
Sep 05 02:40:41.650720 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.650704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:41.660156 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.660133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9c5f057c"]
Sep 05 02:40:41.779354 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.779324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mgh\" (UniqueName: \"kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh\") pod \"curl-metrics-9c5f057c\" (UID: \"0168f12c-88c1-478e-8b0c-439867ccc615\") " pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:41.880032 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.879996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mgh\" (UniqueName: \"kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh\") pod \"curl-metrics-9c5f057c\" (UID: \"0168f12c-88c1-478e-8b0c-439867ccc615\") " pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:41.888813 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.888791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mgh\" (UniqueName: \"kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh\") pod \"curl-metrics-9c5f057c\" (UID: \"0168f12c-88c1-478e-8b0c-439867ccc615\") " pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:41.967855 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:41.967792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:42.235780 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:42.235714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9c5f057c"]
Sep 05 02:40:42.237971 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:40:42.237942    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0168f12c_88c1_478e_8b0c_439867ccc615.slice/crio-e2f91e7641ae86ffaa9f3584c5589f9a7fcedc8fb52d4db839646aff67415a4b WatchSource:0}: Error finding container e2f91e7641ae86ffaa9f3584c5589f9a7fcedc8fb52d4db839646aff67415a4b: Status 404 returned error can't find the container with id e2f91e7641ae86ffaa9f3584c5589f9a7fcedc8fb52d4db839646aff67415a4b
Sep 05 02:40:42.239799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:42.239778    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:40:43.023223 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:43.023193    2560 generic.go:356] "Generic (PLEG): container finished" podID="0168f12c-88c1-478e-8b0c-439867ccc615" containerID="6e294e5de77fc83856a28c1e9474a15effb0c8460de738d7d6a752256c3c0dac" exitCode=0
Sep 05 02:40:43.023527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:43.023285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9c5f057c" event={"ID":"0168f12c-88c1-478e-8b0c-439867ccc615","Type":"ContainerDied","Data":"6e294e5de77fc83856a28c1e9474a15effb0c8460de738d7d6a752256c3c0dac"}
Sep 05 02:40:43.023527 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:43.023330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9c5f057c" event={"ID":"0168f12c-88c1-478e-8b0c-439867ccc615","Type":"ContainerStarted","Data":"e2f91e7641ae86ffaa9f3584c5589f9a7fcedc8fb52d4db839646aff67415a4b"}
Sep 05 02:40:44.151513 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.151490    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:44.187319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.187292    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-9c5f057c_0168f12c-88c1-478e-8b0c-439867ccc615/curl-metrics-9c5f057c/0.log"
Sep 05 02:40:44.199560 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.199539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6mgh\" (UniqueName: \"kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh\") pod \"0168f12c-88c1-478e-8b0c-439867ccc615\" (UID: \"0168f12c-88c1-478e-8b0c-439867ccc615\") "
Sep 05 02:40:44.201426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.201404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh" (OuterVolumeSpecName: "kube-api-access-h6mgh") pod "0168f12c-88c1-478e-8b0c-439867ccc615" (UID: "0168f12c-88c1-478e-8b0c-439867ccc615"). InnerVolumeSpecName "kube-api-access-h6mgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:44.208824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.208803    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-9c5f057c"]
Sep 05 02:40:44.212389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.212366    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-9c5f057c"]
Sep 05 02:40:44.300616 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.300553    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6mgh\" (UniqueName: \"kubernetes.io/projected/0168f12c-88c1-478e-8b0c-439867ccc615-kube-api-access-h6mgh\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:44.969649 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:44.969614    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0168f12c-88c1-478e-8b0c-439867ccc615" path="/var/lib/kubelet/pods/0168f12c-88c1-478e-8b0c-439867ccc615/volumes"
Sep 05 02:40:45.032974 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:45.032933    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9c5f057c"
Sep 05 02:40:45.033099 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:45.032984    2560 scope.go:117] "RemoveContainer" containerID="6e294e5de77fc83856a28c1e9474a15effb0c8460de738d7d6a752256c3c0dac"
Sep 05 02:40:49.597892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.597860    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-b910b152"]
Sep 05 02:40:49.598242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.598177    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0168f12c-88c1-478e-8b0c-439867ccc615" containerName="curl-metrics-9c5f057c"
Sep 05 02:40:49.598242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.598188    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0168f12c-88c1-478e-8b0c-439867ccc615" containerName="curl-metrics-9c5f057c"
Sep 05 02:40:49.598311 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.598252    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0168f12c-88c1-478e-8b0c-439867ccc615" containerName="curl-metrics-9c5f057c"
Sep 05 02:40:49.602388 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.602369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:49.607153 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.607127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b910b152"]
Sep 05 02:40:49.640931 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.640910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n299w\" (UniqueName: \"kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w\") pod \"curl-metrics-b910b152\" (UID: \"c7d9240c-6753-41a9-803d-81ae86fab440\") " pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:49.742263 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.742239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n299w\" (UniqueName: \"kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w\") pod \"curl-metrics-b910b152\" (UID: \"c7d9240c-6753-41a9-803d-81ae86fab440\") " pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:49.750396 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.750379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n299w\" (UniqueName: \"kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w\") pod \"curl-metrics-b910b152\" (UID: \"c7d9240c-6753-41a9-803d-81ae86fab440\") " pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:49.924054 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:49.924029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:50.192815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:50.192722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b910b152"]
Sep 05 02:40:50.195585 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:40:50.195557    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d9240c_6753_41a9_803d_81ae86fab440.slice/crio-fd89cf3574cefc133380469526ff3779694c81042b1681f3663db7c3164eecea WatchSource:0}: Error finding container fd89cf3574cefc133380469526ff3779694c81042b1681f3663db7c3164eecea: Status 404 returned error can't find the container with id fd89cf3574cefc133380469526ff3779694c81042b1681f3663db7c3164eecea
Sep 05 02:40:51.063197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:51.063114    2560 generic.go:356] "Generic (PLEG): container finished" podID="c7d9240c-6753-41a9-803d-81ae86fab440" containerID="c284f712c3aca7fa93da147ae7ea7eac18f5a24bc859646a7e8722c8ad23ab3a" exitCode=0
Sep 05 02:40:51.063197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:51.063177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b910b152" event={"ID":"c7d9240c-6753-41a9-803d-81ae86fab440","Type":"ContainerDied","Data":"c284f712c3aca7fa93da147ae7ea7eac18f5a24bc859646a7e8722c8ad23ab3a"}
Sep 05 02:40:51.063558 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:51.063212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b910b152" event={"ID":"c7d9240c-6753-41a9-803d-81ae86fab440","Type":"ContainerStarted","Data":"fd89cf3574cefc133380469526ff3779694c81042b1681f3663db7c3164eecea"}
Sep 05 02:40:52.186228 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.186206    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:52.204655 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.204636    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-b910b152_c7d9240c-6753-41a9-803d-81ae86fab440/curl-metrics-b910b152/0.log"
Sep 05 02:40:52.227487 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.227462    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-b910b152"]
Sep 05 02:40:52.231187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.231168    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-b910b152"]
Sep 05 02:40:52.258957 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.258933    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n299w\" (UniqueName: \"kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w\") pod \"c7d9240c-6753-41a9-803d-81ae86fab440\" (UID: \"c7d9240c-6753-41a9-803d-81ae86fab440\") "
Sep 05 02:40:52.260863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.260844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w" (OuterVolumeSpecName: "kube-api-access-n299w") pod "c7d9240c-6753-41a9-803d-81ae86fab440" (UID: "c7d9240c-6753-41a9-803d-81ae86fab440"). InnerVolumeSpecName "kube-api-access-n299w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:52.360480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.360416    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n299w\" (UniqueName: \"kubernetes.io/projected/c7d9240c-6753-41a9-803d-81ae86fab440-kube-api-access-n299w\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:40:52.970359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:52.970315    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d9240c-6753-41a9-803d-81ae86fab440" path="/var/lib/kubelet/pods/c7d9240c-6753-41a9-803d-81ae86fab440/volumes"
Sep 05 02:40:53.072281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:53.072255    2560 scope.go:117] "RemoveContainer" containerID="c284f712c3aca7fa93da147ae7ea7eac18f5a24bc859646a7e8722c8ad23ab3a"
Sep 05 02:40:53.072391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:53.072286    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b910b152"
Sep 05 02:40:57.606403 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.606364    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:40:57.606971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.606896    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d9240c-6753-41a9-803d-81ae86fab440" containerName="curl-metrics-b910b152"
Sep 05 02:40:57.606971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.606916    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d9240c-6753-41a9-803d-81ae86fab440" containerName="curl-metrics-b910b152"
Sep 05 02:40:57.607107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.606988    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d9240c-6753-41a9-803d-81ae86fab440" containerName="curl-metrics-b910b152"
Sep 05 02:40:57.611346 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.611323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.613927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.613906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:40:57.614027 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.614013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 05 02:40:57.614096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.614038    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:40:57.615073 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.615056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:40:57.615073 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.615057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 05 02:40:57.615073 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.615057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:40:57.615236 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.615109    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:40:57.620532 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.620511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:40:57.703795 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.703761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.703950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.703805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.703950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.703842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.703950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.703870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.704095 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.703976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9xd\" (UniqueName: \"kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805218 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9xd\" (UniqueName: \"kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805801 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.805984 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.805978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.807734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.807716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.813380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.813353    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9xd\" (UniqueName: \"kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd\") pod \"isvc-sklearn-predictor-758497cd99-p2lmm\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:57.922066 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:57.922043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:40:58.194103 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:58.194051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:40:58.196209 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:40:58.196186    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0275b123_0c2b_4294_b7f4_d6b7e62a811e.slice/crio-d34312107647c582095a44b255a5c99afe3c2e2fd281245aeaa45293eb4af378 WatchSource:0}: Error finding container d34312107647c582095a44b255a5c99afe3c2e2fd281245aeaa45293eb4af378: Status 404 returned error can't find the container with id d34312107647c582095a44b255a5c99afe3c2e2fd281245aeaa45293eb4af378
Sep 05 02:40:59.100450 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:59.100415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerStarted","Data":"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"}
Sep 05 02:40:59.100811 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:40:59.100479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerStarted","Data":"d34312107647c582095a44b255a5c99afe3c2e2fd281245aeaa45293eb4af378"}
Sep 05 02:41:02.114613 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:02.114576    2560 generic.go:356] "Generic (PLEG): container finished" podID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerID="baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d" exitCode=0
Sep 05 02:41:02.114955 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:02.114649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerDied","Data":"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"}
Sep 05 02:41:03.121302 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.121261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerStarted","Data":"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"}
Sep 05 02:41:03.121713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.121320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerStarted","Data":"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"}
Sep 05 02:41:03.121713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.121582    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:41:03.121820 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.121719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:41:03.123015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.122984    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:03.142525 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:03.142477    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podStartSLOduration=2.49549919 podStartE2EDuration="6.142429067s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.646929877s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:03.141210581 +0000 UTC m=+1406.780422403" watchObservedRunningTime="2026-09-05 02:41:03.142429067 +0000 UTC m=+1406.781640866"
Sep 05 02:41:04.126254 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:04.126206    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:09.130356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:09.130322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:41:09.130884 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:09.130856    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:19.131385 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:19.131345    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:29.130938 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:29.130852    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:39.131260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:39.131221    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:49.130993 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:49.130950    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:41:59.130986 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:41:59.130940    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:09.131657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:09.131615    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:19.131906 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:19.131871    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:42:27.713615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:27.713571    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:42:27.714126 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:27.713835    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" containerID="cri-o://23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030" gracePeriod=30
Sep 05 02:42:27.714126 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:27.713867    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kube-rbac-proxy" containerID="cri-o://c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a" gracePeriod=30
Sep 05 02:42:28.494444 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:28.494411    2560 generic.go:356] "Generic (PLEG): container finished" podID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerID="c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a" exitCode=2
Sep 05 02:42:28.494623 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:28.494482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerDied","Data":"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"}
Sep 05 02:42:29.126543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:29.126497    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 05 02:42:29.131497 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:29.131470    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:32.048617 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.048594    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:42:32.195669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.195645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg9xd\" (UniqueName: \"kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd\") pod \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") "
Sep 05 02:42:32.195807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.195686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert\") pod \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") "
Sep 05 02:42:32.195807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.195715    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls\") pod \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") "
Sep 05 02:42:32.195807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.195750    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config\") pod \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") "
Sep 05 02:42:32.195807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.195782    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location\") pod \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\" (UID: \"0275b123-0c2b-4294-b7f4-d6b7e62a811e\") "
Sep 05 02:42:32.196161 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.196137    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0275b123-0c2b-4294-b7f4-d6b7e62a811e" (UID: "0275b123-0c2b-4294-b7f4-d6b7e62a811e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:32.196248 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.196179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0275b123-0c2b-4294-b7f4-d6b7e62a811e" (UID: "0275b123-0c2b-4294-b7f4-d6b7e62a811e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:32.196248 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.196201    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0275b123-0c2b-4294-b7f4-d6b7e62a811e" (UID: "0275b123-0c2b-4294-b7f4-d6b7e62a811e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:32.197784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.197762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0275b123-0c2b-4294-b7f4-d6b7e62a811e" (UID: "0275b123-0c2b-4294-b7f4-d6b7e62a811e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:32.197840 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.197778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd" (OuterVolumeSpecName: "kube-api-access-jg9xd") pod "0275b123-0c2b-4294-b7f4-d6b7e62a811e" (UID: "0275b123-0c2b-4294-b7f4-d6b7e62a811e"). InnerVolumeSpecName "kube-api-access-jg9xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:32.296866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.296841    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9xd\" (UniqueName: \"kubernetes.io/projected/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-api-access-jg9xd\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:42:32.296866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.296863    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:42:32.296990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.296872    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0275b123-0c2b-4294-b7f4-d6b7e62a811e-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:42:32.296990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.296880    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:42:32.296990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.296889    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0275b123-0c2b-4294-b7f4-d6b7e62a811e-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:42:32.512772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.512690    2560 generic.go:356] "Generic (PLEG): container finished" podID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerID="23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030" exitCode=0
Sep 05 02:42:32.512772 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.512762    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"
Sep 05 02:42:32.512907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.512779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerDied","Data":"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"}
Sep 05 02:42:32.512907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.512839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm" event={"ID":"0275b123-0c2b-4294-b7f4-d6b7e62a811e","Type":"ContainerDied","Data":"d34312107647c582095a44b255a5c99afe3c2e2fd281245aeaa45293eb4af378"}
Sep 05 02:42:32.512907 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.512868    2560 scope.go:117] "RemoveContainer" containerID="c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"
Sep 05 02:42:32.521969 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.521948    2560 scope.go:117] "RemoveContainer" containerID="23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"
Sep 05 02:42:32.529462 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.529444    2560 scope.go:117] "RemoveContainer" containerID="baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"
Sep 05 02:42:32.535388 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.535364    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:42:32.537786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.537764    2560 scope.go:117] "RemoveContainer" containerID="c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"
Sep 05 02:42:32.538038 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:42:32.538023    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a\": container with ID starting with c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a not found: ID does not exist" containerID="c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"
Sep 05 02:42:32.538102 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538044    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a"} err="failed to get container status \"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a\": rpc error: code = NotFound desc = could not find container \"c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a\": container with ID starting with c2ea345b72f3edf65bae82582d8db2e5a8bd97c1848680456a4fb872e0b5a41a not found: ID does not exist"
Sep 05 02:42:32.538102 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538058    2560 scope.go:117] "RemoveContainer" containerID="23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"
Sep 05 02:42:32.538312 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:42:32.538294    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030\": container with ID starting with 23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030 not found: ID does not exist" containerID="23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"
Sep 05 02:42:32.538366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030"} err="failed to get container status \"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030\": rpc error: code = NotFound desc = could not find container \"23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030\": container with ID starting with 23753c22dfb8618c51d70343e1c9cbfdc7d8dd1ee31c74bbcbd0ea30a804e030 not found: ID does not exist"
Sep 05 02:42:32.538366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538332    2560 scope.go:117] "RemoveContainer" containerID="baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"
Sep 05 02:42:32.538366 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538333    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-758497cd99-p2lmm"]
Sep 05 02:42:32.538567 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:42:32.538551    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d\": container with ID starting with baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d not found: ID does not exist" containerID="baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"
Sep 05 02:42:32.538607 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.538571    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d"} err="failed to get container status \"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d\": rpc error: code = NotFound desc = could not find container \"baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d\": container with ID starting with baf9bf5d81d5193a2e43a0952c89421648e54ed4a856b9bde690967b4528515d not found: ID does not exist"
Sep 05 02:42:32.884829 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.884801    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:42:32.885271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885258    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container"
Sep 05 02:42:32.885315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885275    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container"
Sep 05 02:42:32.885315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885290    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kube-rbac-proxy"
Sep 05 02:42:32.885315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885299    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kube-rbac-proxy"
Sep 05 02:42:32.885408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885314    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="storage-initializer"
Sep 05 02:42:32.885408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885323    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="storage-initializer"
Sep 05 02:42:32.885408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885386    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kube-rbac-proxy"
Sep 05 02:42:32.885408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.885398    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" containerName="kserve-container"
Sep 05 02:42:32.890065 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.890041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:32.892539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.892517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 05 02:42:32.892666 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.892644    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:42:32.892733 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.892663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:42:32.892849 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.892828    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:42:32.893780 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.893761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 05 02:42:32.893867 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.893770    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:42:32.893867 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.893858    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:42:32.897926 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.897904    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:42:32.969729 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:32.969695    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0275b123-0c2b-4294-b7f4-d6b7e62a811e" path="/var/lib/kubelet/pods/0275b123-0c2b-4294-b7f4-d6b7e62a811e/volumes"
Sep 05 02:42:33.001994 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.001971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.002068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.002018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.002068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.002041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.002151 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.002079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjhg\" (UniqueName: \"kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.002151 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.002100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.102515 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.102515 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjhg\" (UniqueName: \"kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103042 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.102968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.103371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.103431 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.103391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.104865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.104847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.110839 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.110814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjhg\" (UniqueName: \"kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.202887 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.202828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:42:33.473233 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.473199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:42:33.475419 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:42:33.475392    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde26c756_aed4_4f73_8bc5_dbdc6aa72aa4.slice/crio-25492f49314db600151e8277dd8cc2ab4f6d8809dfbf279236bfb1a1fba3caa4 WatchSource:0}: Error finding container 25492f49314db600151e8277dd8cc2ab4f6d8809dfbf279236bfb1a1fba3caa4: Status 404 returned error can't find the container with id 25492f49314db600151e8277dd8cc2ab4f6d8809dfbf279236bfb1a1fba3caa4
Sep 05 02:42:33.519148 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:33.519122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerStarted","Data":"25492f49314db600151e8277dd8cc2ab4f6d8809dfbf279236bfb1a1fba3caa4"}
Sep 05 02:42:34.525515 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:34.525479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerStarted","Data":"386364d0550eb0e52e6b88f8445ae02a2e5acec8ee7dec717675b0e0e2bc1ed5"}
Sep 05 02:42:36.854438 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:36.854406    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:42:36.859025 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:36.859004    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:42:37.541675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:37.541593    2560 generic.go:356] "Generic (PLEG): container finished" podID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerID="386364d0550eb0e52e6b88f8445ae02a2e5acec8ee7dec717675b0e0e2bc1ed5" exitCode=0
Sep 05 02:42:37.541675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:42:37.541634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerDied","Data":"386364d0550eb0e52e6b88f8445ae02a2e5acec8ee7dec717675b0e0e2bc1ed5"}
Sep 05 02:44:38.148305 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:38.148280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerStarted","Data":"c96c39f6325fc0fcfc07d1008d80646ffd8a4e01ae96fbe6038a1db9c212cf39"}
Sep 05 02:44:39.154131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:39.154097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerStarted","Data":"22e012f6e6427489d50bf33450e801b3e9f9fd46052432542d68437498ff518b"}
Sep 05 02:44:39.154493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:39.154313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:44:39.154493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:39.154344    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:44:39.188197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:39.188144    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" podStartSLOduration=3.069925985 podStartE2EDuration="2m7.1881289s" totalImagesPullingTime="2m0.417587138s" totalInitContainerRuntime="3.700615777s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:44:39.187818726 +0000 UTC m=+1622.827030555" watchObservedRunningTime="2026-09-05 02:44:39.1881289 +0000 UTC m=+1622.827340700"
Sep 05 02:44:45.164394 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:44:45.164364    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:45:15.231997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:15.231941    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 05 02:45:25.167058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:25.167024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:45:33.002665 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:33.002633    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:45:33.003162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:33.002938    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kserve-container" containerID="cri-o://c96c39f6325fc0fcfc07d1008d80646ffd8a4e01ae96fbe6038a1db9c212cf39" gracePeriod=30
Sep 05 02:45:33.003162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:33.002984    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kube-rbac-proxy" containerID="cri-o://22e012f6e6427489d50bf33450e801b3e9f9fd46052432542d68437498ff518b" gracePeriod=30
Sep 05 02:45:33.377229 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:33.377152    2560 generic.go:356] "Generic (PLEG): container finished" podID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerID="22e012f6e6427489d50bf33450e801b3e9f9fd46052432542d68437498ff518b" exitCode=2
Sep 05 02:45:33.377229 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:33.377205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerDied","Data":"22e012f6e6427489d50bf33450e801b3e9f9fd46052432542d68437498ff518b"}
Sep 05 02:45:35.159354 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:35.159313    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 05 02:45:38.185373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.185336    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:45:38.187946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.187923    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.190330 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.190307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 05 02:45:38.190409 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.190369    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:45:38.199295 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.199275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:45:38.280788 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.280763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.280970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.280844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.280970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.280873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wctr\" (UniqueName: \"kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.280970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.280929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.280970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.280956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.381755 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.381718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.381867 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.381777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wctr\" (UniqueName: \"kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.381946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.381927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.382001 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.381969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.382082 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:45:38.382068    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 05 02:45:38.382135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.382082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.382135 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:45:38.382126    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls podName:902fc3dc-d29d-47a5-b444-2974bc0fee37 nodeName:}" failed. No retries permitted until 2026-09-05 02:45:38.882110474 +0000 UTC m=+1682.521322252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls") pod "isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 05 02:45:38.382334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.382314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.382459 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.382440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.382603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.382586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.390410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.390387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wctr\" (UniqueName: \"kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.886563 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.886529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:38.888933 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:38.888911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:39.097945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.097915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:39.370723 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.370694    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:45:39.371193 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:45:39.371168    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902fc3dc_d29d_47a5_b444_2974bc0fee37.slice/crio-1a1244f1a6dd40acc4f0a2791ad6465c28259957a6487a5e17ee6e6c3ca98e18 WatchSource:0}: Error finding container 1a1244f1a6dd40acc4f0a2791ad6465c28259957a6487a5e17ee6e6c3ca98e18: Status 404 returned error can't find the container with id 1a1244f1a6dd40acc4f0a2791ad6465c28259957a6487a5e17ee6e6c3ca98e18
Sep 05 02:45:39.405825 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.405758    2560 generic.go:356] "Generic (PLEG): container finished" podID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerID="c96c39f6325fc0fcfc07d1008d80646ffd8a4e01ae96fbe6038a1db9c212cf39" exitCode=0
Sep 05 02:45:39.405825 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.405774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerDied","Data":"c96c39f6325fc0fcfc07d1008d80646ffd8a4e01ae96fbe6038a1db9c212cf39"}
Sep 05 02:45:39.406941 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.406923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerStarted","Data":"1a1244f1a6dd40acc4f0a2791ad6465c28259957a6487a5e17ee6e6c3ca98e18"}
Sep 05 02:45:39.442262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.442244    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:45:39.593753 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.593718    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location\") pod \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") "
Sep 05 02:45:39.593901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.593800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls\") pod \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") "
Sep 05 02:45:39.593901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.593818    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert\") pod \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") "
Sep 05 02:45:39.593901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.593853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config\") pod \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") "
Sep 05 02:45:39.594100 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594030    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjhg\" (UniqueName: \"kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg\") pod \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\" (UID: \"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4\") "
Sep 05 02:45:39.594159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594095    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" (UID: "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:39.594217 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" (UID: "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:39.594217 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594180    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" (UID: "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:39.594379 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594359    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:45:39.594438 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594385    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:45:39.594438 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.594401    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:45:39.595910 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.595882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" (UID: "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:39.596050 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.595957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg" (OuterVolumeSpecName: "kube-api-access-pvjhg") pod "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" (UID: "de26c756-aed4-4f73-8bc5-dbdc6aa72aa4"). InnerVolumeSpecName "kube-api-access-pvjhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:39.695812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.695719    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:45:39.695812 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:39.695761    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjhg\" (UniqueName: \"kubernetes.io/projected/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4-kube-api-access-pvjhg\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:45:40.413024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.412981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerStarted","Data":"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"}
Sep 05 02:45:40.415032 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.415004    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8" event={"ID":"de26c756-aed4-4f73-8bc5-dbdc6aa72aa4","Type":"ContainerDied","Data":"25492f49314db600151e8277dd8cc2ab4f6d8809dfbf279236bfb1a1fba3caa4"}
Sep 05 02:45:40.415184 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.415041    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"
Sep 05 02:45:40.415184 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.415067    2560 scope.go:117] "RemoveContainer" containerID="22e012f6e6427489d50bf33450e801b3e9f9fd46052432542d68437498ff518b"
Sep 05 02:45:40.424285 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.424267    2560 scope.go:117] "RemoveContainer" containerID="c96c39f6325fc0fcfc07d1008d80646ffd8a4e01ae96fbe6038a1db9c212cf39"
Sep 05 02:45:40.431483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.431467    2560 scope.go:117] "RemoveContainer" containerID="386364d0550eb0e52e6b88f8445ae02a2e5acec8ee7dec717675b0e0e2bc1ed5"
Sep 05 02:45:40.454400 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.454377    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:45:40.456547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.456526    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7nqk8"]
Sep 05 02:45:40.969724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:40.969688    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" path="/var/lib/kubelet/pods/de26c756-aed4-4f73-8bc5-dbdc6aa72aa4/volumes"
Sep 05 02:45:46.443279 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:46.443244    2560 generic.go:356] "Generic (PLEG): container finished" podID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerID="db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec" exitCode=0
Sep 05 02:45:46.443697 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:46.443325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerDied","Data":"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"}
Sep 05 02:45:47.448844 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.448812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerStarted","Data":"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"}
Sep 05 02:45:47.449242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.448871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerStarted","Data":"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"}
Sep 05 02:45:47.449242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.449151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:47.449371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.449320    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:47.450619 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.450597    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:45:47.468638 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:47.468596    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podStartSLOduration=2.668692547 podStartE2EDuration="9.468584205s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.799891658s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:45:47.467011637 +0000 UTC m=+1691.106223438" watchObservedRunningTime="2026-09-05 02:45:47.468584205 +0000 UTC m=+1691.107796004"
Sep 05 02:45:48.453844 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:48.453800    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:45:49.457760 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:49.457711    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:45:54.462145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:54.462115    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:45:54.462643 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:45:54.462613    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:46:04.463333 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:04.463261    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:46:07.576699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.576667    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kserve-container"
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577019    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kserve-container"
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577032    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="storage-initializer"
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577038    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="storage-initializer"
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577045    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kube-rbac-proxy"
Sep 05 02:46:07.577101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577051    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kube-rbac-proxy"
Sep 05 02:46:07.577326 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577113    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kube-rbac-proxy"
Sep 05 02:46:07.577326 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.577120    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="de26c756-aed4-4f73-8bc5-dbdc6aa72aa4" containerName="kserve-container"
Sep 05 02:46:07.579190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.579171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.581698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.581676    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 05 02:46:07.581823 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.581685    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 05 02:46:07.590570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.590550    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:46:07.729566 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.729531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.729739 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.729596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.729739 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.729649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8r4\" (UniqueName: \"kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.729739 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.729705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.729739 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.729734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.830716 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.830641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8r4\" (UniqueName: \"kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.830716 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.830684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.830716 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.830704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.830985 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.830761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.830985 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.830796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.831310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.831286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.831509 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.831491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.831718 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.831694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.833037 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.833018    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.838721 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.838695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8r4\" (UniqueName: \"kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4\") pod \"isvc-canary-create-predictor-54dc6cb966-gmtqr\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:07.889685 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:07.889659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:08.163629 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:08.163477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:46:08.165793 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:46:08.165759    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb30d479_577c_497f_a523_533decce2f1a.slice/crio-65c066c6b0bea5d547d0fa0f8d7ec2eea6434735029cad9d340310e95c428b80 WatchSource:0}: Error finding container 65c066c6b0bea5d547d0fa0f8d7ec2eea6434735029cad9d340310e95c428b80: Status 404 returned error can't find the container with id 65c066c6b0bea5d547d0fa0f8d7ec2eea6434735029cad9d340310e95c428b80
Sep 05 02:46:08.550189 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:08.550094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerStarted","Data":"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"}
Sep 05 02:46:08.550189 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:08.550154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerStarted","Data":"65c066c6b0bea5d547d0fa0f8d7ec2eea6434735029cad9d340310e95c428b80"}
Sep 05 02:46:10.092823 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:10.092790    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:46:10.093309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:10.093009    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container" containerID="cri-o://c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b" gracePeriod=30
Sep 05 02:46:10.093309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:10.093057    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kube-rbac-proxy" containerID="cri-o://0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd" gracePeriod=30
Sep 05 02:46:10.562829 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:10.562795    2560 generic.go:356] "Generic (PLEG): container finished" podID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerID="0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd" exitCode=2
Sep 05 02:46:10.562992 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:10.562862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerDied","Data":"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"}
Sep 05 02:46:11.033372 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.033351    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:46:11.163757 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.163723    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") pod \"902fc3dc-d29d-47a5-b444-2974bc0fee37\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") "
Sep 05 02:46:11.164112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.163833    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wctr\" (UniqueName: \"kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr\") pod \"902fc3dc-d29d-47a5-b444-2974bc0fee37\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") "
Sep 05 02:46:11.164112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.163867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert\") pod \"902fc3dc-d29d-47a5-b444-2974bc0fee37\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") "
Sep 05 02:46:11.164112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.163899    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location\") pod \"902fc3dc-d29d-47a5-b444-2974bc0fee37\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") "
Sep 05 02:46:11.164112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.163932    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config\") pod \"902fc3dc-d29d-47a5-b444-2974bc0fee37\" (UID: \"902fc3dc-d29d-47a5-b444-2974bc0fee37\") "
Sep 05 02:46:11.164335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.164288    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "902fc3dc-d29d-47a5-b444-2974bc0fee37" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:46:11.164389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.164331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "902fc3dc-d29d-47a5-b444-2974bc0fee37" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:46:11.165831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.165804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "902fc3dc-d29d-47a5-b444-2974bc0fee37" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:46:11.165937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.165867    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr" (OuterVolumeSpecName: "kube-api-access-6wctr") pod "902fc3dc-d29d-47a5-b444-2974bc0fee37" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37"). InnerVolumeSpecName "kube-api-access-6wctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:46:11.192354 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.192330    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "902fc3dc-d29d-47a5-b444-2974bc0fee37" (UID: "902fc3dc-d29d-47a5-b444-2974bc0fee37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:46:11.264822 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.264799    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902fc3dc-d29d-47a5-b444-2974bc0fee37-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:46:11.264822 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.264820    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wctr\" (UniqueName: \"kubernetes.io/projected/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-api-access-6wctr\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:46:11.264954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.264830    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:46:11.264954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.264839    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902fc3dc-d29d-47a5-b444-2974bc0fee37-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:46:11.264954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.264848    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902fc3dc-d29d-47a5-b444-2974bc0fee37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:46:11.569956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.569879    2560 generic.go:356] "Generic (PLEG): container finished" podID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerID="c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b" exitCode=0
Sep 05 02:46:11.569956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.569947    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"
Sep 05 02:46:11.570115 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.569960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerDied","Data":"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"}
Sep 05 02:46:11.570115 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.570014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw" event={"ID":"902fc3dc-d29d-47a5-b444-2974bc0fee37","Type":"ContainerDied","Data":"1a1244f1a6dd40acc4f0a2791ad6465c28259957a6487a5e17ee6e6c3ca98e18"}
Sep 05 02:46:11.570115 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.570040    2560 scope.go:117] "RemoveContainer" containerID="0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"
Sep 05 02:46:11.579416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.579399    2560 scope.go:117] "RemoveContainer" containerID="c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"
Sep 05 02:46:11.586908 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.586887    2560 scope.go:117] "RemoveContainer" containerID="db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"
Sep 05 02:46:11.595506 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.595486    2560 scope.go:117] "RemoveContainer" containerID="0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"
Sep 05 02:46:11.595807 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:46:11.595788    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd\": container with ID starting with 0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd not found: ID does not exist" containerID="0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"
Sep 05 02:46:11.595901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.595818    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd"} err="failed to get container status \"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd\": rpc error: code = NotFound desc = could not find container \"0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd\": container with ID starting with 0d54a2dce55246aebdd228fd4450827bc15f88c10bee10d3d9de8bda0710eecd not found: ID does not exist"
Sep 05 02:46:11.595901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.595842    2560 scope.go:117] "RemoveContainer" containerID="c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"
Sep 05 02:46:11.596111 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:46:11.596092    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b\": container with ID starting with c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b not found: ID does not exist" containerID="c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"
Sep 05 02:46:11.596166 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.596116    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b"} err="failed to get container status \"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b\": rpc error: code = NotFound desc = could not find container \"c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b\": container with ID starting with c0660fd8f51bc708b6d09cad1cfaea95e085ca59ce5897bdb76fc5c0e5386c9b not found: ID does not exist"
Sep 05 02:46:11.596166 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.596131    2560 scope.go:117] "RemoveContainer" containerID="db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"
Sep 05 02:46:11.596374 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:46:11.596359    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec\": container with ID starting with db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec not found: ID does not exist" containerID="db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"
Sep 05 02:46:11.596431 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.596380    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec"} err="failed to get container status \"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec\": rpc error: code = NotFound desc = could not find container \"db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec\": container with ID starting with db4f5493a1d9279ebbcb419dc8c428982f9c851c54957a99e705f3919cdee5ec not found: ID does not exist"
Sep 05 02:46:11.605249 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.605218    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:46:11.606159 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:46:11.606140    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902fc3dc_d29d_47a5_b444_2974bc0fee37.slice\": RecentStats: unable to find data in memory cache]"
Sep 05 02:46:11.610252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:11.610231    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fdc964fbc-zs5sw"]
Sep 05 02:46:12.576507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:12.576476    2560 generic.go:356] "Generic (PLEG): container finished" podID="fb30d479-577c-497f-a523-533decce2f1a" containerID="6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62" exitCode=0
Sep 05 02:46:12.576898 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:12.576549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerDied","Data":"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"}
Sep 05 02:46:12.971265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:12.971229    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" path="/var/lib/kubelet/pods/902fc3dc-d29d-47a5-b444-2974bc0fee37/volumes"
Sep 05 02:46:13.584854 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.584817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerStarted","Data":"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"}
Sep 05 02:46:13.585235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.584872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerStarted","Data":"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"}
Sep 05 02:46:13.585235 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.585145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:13.585327 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.585288    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:13.586384 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.586358    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:13.607056 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:13.607017    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podStartSLOduration=2.96968617 podStartE2EDuration="6.60700421s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.63731804s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:13.604770781 +0000 UTC m=+1717.243982597" watchObservedRunningTime="2026-09-05 02:46:13.60700421 +0000 UTC m=+1717.246216027"
Sep 05 02:46:14.590159 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:14.590121    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:15.240737 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.240705    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:46:15.241198 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241182    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kube-rbac-proxy"
Sep 05 02:46:15.241242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241201    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kube-rbac-proxy"
Sep 05 02:46:15.241242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container"
Sep 05 02:46:15.241242 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241225    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container"
Sep 05 02:46:15.241335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241241    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="storage-initializer"
Sep 05 02:46:15.241335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241250    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="storage-initializer"
Sep 05 02:46:15.241335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241322    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kserve-container"
Sep 05 02:46:15.241335 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.241333    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="902fc3dc-d29d-47a5-b444-2974bc0fee37" containerName="kube-rbac-proxy"
Sep 05 02:46:15.244661 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.244644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.247549 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.247527    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 05 02:46:15.247663 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.247597    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:46:15.255079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.255057    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:46:15.398855 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.398823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.399017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.398876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.399017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.398946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49cdr\" (UniqueName: \"kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.399017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.398993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.399017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.399016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500318 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500318 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500318 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500605 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500605 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49cdr\" (UniqueName: \"kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.500738 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.500711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.501087 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.501067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.501137 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.501106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.502777 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.502722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.511927 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.511905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49cdr\" (UniqueName: \"kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.555535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.555510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:15.829760 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:15.829714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:46:15.832003 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:46:15.831974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97868fb_8330_41d3_8c34_2d451a2134f3.slice/crio-b3d5e121604c4f02e1e1db8baeb8c4b58f62694d64d95aebbd2f81123715aa83 WatchSource:0}: Error finding container b3d5e121604c4f02e1e1db8baeb8c4b58f62694d64d95aebbd2f81123715aa83: Status 404 returned error can't find the container with id b3d5e121604c4f02e1e1db8baeb8c4b58f62694d64d95aebbd2f81123715aa83
Sep 05 02:46:16.602245 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:16.602210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerStarted","Data":"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"}
Sep 05 02:46:16.602429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:16.602275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerStarted","Data":"b3d5e121604c4f02e1e1db8baeb8c4b58f62694d64d95aebbd2f81123715aa83"}
Sep 05 02:46:19.594646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:19.594616    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:46:19.595138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:19.595110    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:20.624009 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:20.623974    2560 generic.go:356] "Generic (PLEG): container finished" podID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerID="b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755" exitCode=0
Sep 05 02:46:20.624403 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:20.624023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerDied","Data":"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"}
Sep 05 02:46:21.630268 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:21.630237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerStarted","Data":"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"}
Sep 05 02:46:21.630629 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:21.630293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerStarted","Data":"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"}
Sep 05 02:46:21.630629 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:21.630472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:21.652224 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:21.652177    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podStartSLOduration=2.656150363 podStartE2EDuration="6.652164502s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996014139s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:21.649910636 +0000 UTC m=+1725.289122455" watchObservedRunningTime="2026-09-05 02:46:21.652164502 +0000 UTC m=+1725.291376301"
Sep 05 02:46:22.634836 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:22.634787    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:28.645105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:28.645075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:46:29.595990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:29.595946    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:39.596057 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:39.596010    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:49.595578 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:49.595536    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:46:58.732657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:58.732609    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 05 02:46:59.595265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:46:59.595215    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:47:08.648457 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:08.648425    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:47:09.595176 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:09.595137    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:47:15.269961 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:15.269926    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:47:15.270460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:15.270189    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container" containerID="cri-o://3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184" gracePeriod=30
Sep 05 02:47:15.270460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:15.270225    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kube-rbac-proxy" containerID="cri-o://309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2" gracePeriod=30
Sep 05 02:47:15.879398 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:15.879364    2560 generic.go:356] "Generic (PLEG): container finished" podID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerID="309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2" exitCode=2
Sep 05 02:47:15.879566 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:15.879417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerDied","Data":"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"}
Sep 05 02:47:18.640603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:18.640557    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 05 02:47:18.645996 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:18.645965    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:47:19.595620 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:19.595577    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:47:20.460755 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.460716    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:47:20.463989 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.463970    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.466824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.466804    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 05 02:47:20.466824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.466811    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 05 02:47:20.475285 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.475262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:47:20.626997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.626959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fllm\" (UniqueName: \"kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.626997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.626999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.627182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.627031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.627182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.627057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.627182 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.627072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.727732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.727659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.727732 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.727711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.727953 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.727824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.727953 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.727920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fllm\" (UniqueName: \"kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.728075 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.727978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.728352 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.728326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.728465 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.728448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.728529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.728508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.730079 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.730062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.736439 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.736413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fllm\" (UniqueName: \"kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm\") pod \"isvc-sklearn-v2-predictor-5cffc756c9-lm9bx\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:20.773895 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:20.773875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:21.045781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:21.045762    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:47:21.048485 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:47:21.048452    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9d8777_8b57_45aa_bc2e_266713ca68e9.slice/crio-fe393b96cfbdf506e4380541018f9a2e6497a02aaf1214b90e53f1f595995d4b WatchSource:0}: Error finding container fe393b96cfbdf506e4380541018f9a2e6497a02aaf1214b90e53f1f595995d4b: Status 404 returned error can't find the container with id fe393b96cfbdf506e4380541018f9a2e6497a02aaf1214b90e53f1f595995d4b
Sep 05 02:47:21.910260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:21.910228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerStarted","Data":"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"}
Sep 05 02:47:21.910571 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:21.910285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerStarted","Data":"fe393b96cfbdf506e4380541018f9a2e6497a02aaf1214b90e53f1f595995d4b"}
Sep 05 02:47:22.511719 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.511694    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:47:22.643796 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.643765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location\") pod \"f97868fb-8330-41d3-8c34-2d451a2134f3\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") "
Sep 05 02:47:22.643961 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.643857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49cdr\" (UniqueName: \"kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr\") pod \"f97868fb-8330-41d3-8c34-2d451a2134f3\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") "
Sep 05 02:47:22.643961 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.643900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls\") pod \"f97868fb-8330-41d3-8c34-2d451a2134f3\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") "
Sep 05 02:47:22.644098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.643991    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config\") pod \"f97868fb-8330-41d3-8c34-2d451a2134f3\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") "
Sep 05 02:47:22.644098 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.644016    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert\") pod \"f97868fb-8330-41d3-8c34-2d451a2134f3\" (UID: \"f97868fb-8330-41d3-8c34-2d451a2134f3\") "
Sep 05 02:47:22.644426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.644328    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f97868fb-8330-41d3-8c34-2d451a2134f3" (UID: "f97868fb-8330-41d3-8c34-2d451a2134f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:22.644426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.644392    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f97868fb-8330-41d3-8c34-2d451a2134f3" (UID: "f97868fb-8330-41d3-8c34-2d451a2134f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:47:22.644591 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.644503    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f97868fb-8330-41d3-8c34-2d451a2134f3" (UID: "f97868fb-8330-41d3-8c34-2d451a2134f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:22.645996 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.645968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr" (OuterVolumeSpecName: "kube-api-access-49cdr") pod "f97868fb-8330-41d3-8c34-2d451a2134f3" (UID: "f97868fb-8330-41d3-8c34-2d451a2134f3"). InnerVolumeSpecName "kube-api-access-49cdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:47:22.646096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.646066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f97868fb-8330-41d3-8c34-2d451a2134f3" (UID: "f97868fb-8330-41d3-8c34-2d451a2134f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:47:22.744651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.744621    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49cdr\" (UniqueName: \"kubernetes.io/projected/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-api-access-49cdr\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:22.744651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.744649    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97868fb-8330-41d3-8c34-2d451a2134f3-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:22.744841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.744660    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:22.744841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.744669    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f97868fb-8330-41d3-8c34-2d451a2134f3-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:22.744841 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.744678    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f97868fb-8330-41d3-8c34-2d451a2134f3-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:22.917465 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.917377    2560 generic.go:356] "Generic (PLEG): container finished" podID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerID="3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184" exitCode=0
Sep 05 02:47:22.917465 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.917433    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"
Sep 05 02:47:22.917944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.917461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerDied","Data":"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"}
Sep 05 02:47:22.917944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.917542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms" event={"ID":"f97868fb-8330-41d3-8c34-2d451a2134f3","Type":"ContainerDied","Data":"b3d5e121604c4f02e1e1db8baeb8c4b58f62694d64d95aebbd2f81123715aa83"}
Sep 05 02:47:22.917944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.917568    2560 scope.go:117] "RemoveContainer" containerID="309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"
Sep 05 02:47:22.926239 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.926223    2560 scope.go:117] "RemoveContainer" containerID="3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"
Sep 05 02:47:22.934678 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.934661    2560 scope.go:117] "RemoveContainer" containerID="b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"
Sep 05 02:47:22.940449 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.940425    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:47:22.941898 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.941881    2560 scope.go:117] "RemoveContainer" containerID="309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"
Sep 05 02:47:22.942136 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:22.942120    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2\": container with ID starting with 309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2 not found: ID does not exist" containerID="309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"
Sep 05 02:47:22.942190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.942143    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2"} err="failed to get container status \"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2\": rpc error: code = NotFound desc = could not find container \"309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2\": container with ID starting with 309b52aadd8f38f111dac3e540d82cc11d0d108d4ccefcc71208cd293a34aed2 not found: ID does not exist"
Sep 05 02:47:22.942190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.942158    2560 scope.go:117] "RemoveContainer" containerID="3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"
Sep 05 02:47:22.942385 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:22.942365    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184\": container with ID starting with 3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184 not found: ID does not exist" containerID="3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"
Sep 05 02:47:22.942455 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.942395    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184"} err="failed to get container status \"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184\": rpc error: code = NotFound desc = could not find container \"3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184\": container with ID starting with 3d5677fe5a3c74c5aa27ae01d3502387a8cdff658463f5d34c4f670535e2b184 not found: ID does not exist"
Sep 05 02:47:22.942455 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.942421    2560 scope.go:117] "RemoveContainer" containerID="b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"
Sep 05 02:47:22.942703 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:22.942672    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755\": container with ID starting with b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755 not found: ID does not exist" containerID="b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"
Sep 05 02:47:22.942804 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.942706    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755"} err="failed to get container status \"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755\": rpc error: code = NotFound desc = could not find container \"b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755\": container with ID starting with b083846eca3e08544f977779aa1e73c732bba0a4e085f01691828c5fb5a92755 not found: ID does not exist"
Sep 05 02:47:22.944493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.944474    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qt8ms"]
Sep 05 02:47:22.970538 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:22.970518    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" path="/var/lib/kubelet/pods/f97868fb-8330-41d3-8c34-2d451a2134f3/volumes"
Sep 05 02:47:24.929672 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:24.929637    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerID="fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e" exitCode=0
Sep 05 02:47:24.930047 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:24.929676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerDied","Data":"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"}
Sep 05 02:47:25.937237 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.937205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerStarted","Data":"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"}
Sep 05 02:47:25.937612 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.937269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerStarted","Data":"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"}
Sep 05 02:47:25.937612 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.937540    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:25.937730 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.937686    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:25.939185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.939151    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:47:25.957197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:25.957153    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podStartSLOduration=2.301275688 podStartE2EDuration="5.957140354s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.655864666s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:47:25.955338707 +0000 UTC m=+1789.594550517" watchObservedRunningTime="2026-09-05 02:47:25.957140354 +0000 UTC m=+1789.596352156"
Sep 05 02:47:26.943407 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:26.943371    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:47:29.595824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:29.595733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:47:31.949457 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:31.949425    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:47:31.949932 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:31.949903    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:47:36.884380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:36.884352    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:47:36.890701 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:36.890681    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:47:41.949863 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:41.949819    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:47:47.546029 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:47.545984    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:47:47.546595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:47.546232    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" containerID="cri-o://6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8" gracePeriod=30
Sep 05 02:47:47.546595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:47.546314    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kube-rbac-proxy" containerID="cri-o://5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b" gracePeriod=30
Sep 05 02:47:48.035989 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:48.035948    2560 generic.go:356] "Generic (PLEG): container finished" podID="fb30d479-577c-497f-a523-533decce2f1a" containerID="5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b" exitCode=2
Sep 05 02:47:48.036173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:48.036032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerDied","Data":"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"}
Sep 05 02:47:49.590799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:49.590755    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 05 02:47:49.595113 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:49.595080    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:47:51.891667 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.891645    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:47:51.949909 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.949870    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:47:51.975893 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.975871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls\") pod \"fb30d479-577c-497f-a523-533decce2f1a\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") "
Sep 05 02:47:51.975998 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.975985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd8r4\" (UniqueName: \"kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4\") pod \"fb30d479-577c-497f-a523-533decce2f1a\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") "
Sep 05 02:47:51.976053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location\") pod \"fb30d479-577c-497f-a523-533decce2f1a\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") "
Sep 05 02:47:51.976053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config\") pod \"fb30d479-577c-497f-a523-533decce2f1a\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") "
Sep 05 02:47:51.976157 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976073    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert\") pod \"fb30d479-577c-497f-a523-533decce2f1a\" (UID: \"fb30d479-577c-497f-a523-533decce2f1a\") "
Sep 05 02:47:51.976408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb30d479-577c-497f-a523-533decce2f1a" (UID: "fb30d479-577c-497f-a523-533decce2f1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:47:51.976508 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976452    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb30d479-577c-497f-a523-533decce2f1a" (UID: "fb30d479-577c-497f-a523-533decce2f1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:51.976508 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.976484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb30d479-577c-497f-a523-533decce2f1a" (UID: "fb30d479-577c-497f-a523-533decce2f1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:51.977971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.977948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4" (OuterVolumeSpecName: "kube-api-access-nd8r4") pod "fb30d479-577c-497f-a523-533decce2f1a" (UID: "fb30d479-577c-497f-a523-533decce2f1a"). InnerVolumeSpecName "kube-api-access-nd8r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:47:51.978080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:51.978004    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb30d479-577c-497f-a523-533decce2f1a" (UID: "fb30d479-577c-497f-a523-533decce2f1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:47:52.056310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.056251    2560 generic.go:356] "Generic (PLEG): container finished" podID="fb30d479-577c-497f-a523-533decce2f1a" containerID="6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8" exitCode=0
Sep 05 02:47:52.056416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.056328    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"
Sep 05 02:47:52.056416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.056333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerDied","Data":"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"}
Sep 05 02:47:52.056416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.056389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr" event={"ID":"fb30d479-577c-497f-a523-533decce2f1a","Type":"ContainerDied","Data":"65c066c6b0bea5d547d0fa0f8d7ec2eea6434735029cad9d340310e95c428b80"}
Sep 05 02:47:52.056416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.056415    2560 scope.go:117] "RemoveContainer" containerID="5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"
Sep 05 02:47:52.065227 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.065205    2560 scope.go:117] "RemoveContainer" containerID="6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"
Sep 05 02:47:52.072565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.072551    2560 scope.go:117] "RemoveContainer" containerID="6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"
Sep 05 02:47:52.076828 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.076808    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8r4\" (UniqueName: \"kubernetes.io/projected/fb30d479-577c-497f-a523-533decce2f1a-kube-api-access-nd8r4\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.076828 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.076827    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30d479-577c-497f-a523-533decce2f1a-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.076997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.076837    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.076997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.076846    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb30d479-577c-497f-a523-533decce2f1a-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.076997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.076855    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30d479-577c-497f-a523-533decce2f1a-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.079949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.079929    2560 scope.go:117] "RemoveContainer" containerID="5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"
Sep 05 02:47:52.080227 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:52.080209    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b\": container with ID starting with 5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b not found: ID does not exist" containerID="5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"
Sep 05 02:47:52.080281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080236    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b"} err="failed to get container status \"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b\": rpc error: code = NotFound desc = could not find container \"5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b\": container with ID starting with 5ab856305fe44b3592494475fb08d8acb09bf9e6c40da51f4f60d307f9cba70b not found: ID does not exist"
Sep 05 02:47:52.080281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080253    2560 scope.go:117] "RemoveContainer" containerID="6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"
Sep 05 02:47:52.080391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080277    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:47:52.080536 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:52.080520    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8\": container with ID starting with 6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8 not found: ID does not exist" containerID="6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"
Sep 05 02:47:52.080581 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080541    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8"} err="failed to get container status \"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8\": rpc error: code = NotFound desc = could not find container \"6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8\": container with ID starting with 6966baa28c79c6225d3c921728998006c7c16da2b5b0b30f562da6ee574cf1a8 not found: ID does not exist"
Sep 05 02:47:52.080581 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080555    2560 scope.go:117] "RemoveContainer" containerID="6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"
Sep 05 02:47:52.080860 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:47:52.080837    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62\": container with ID starting with 6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62 not found: ID does not exist" containerID="6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"
Sep 05 02:47:52.080945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.080864    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62"} err="failed to get container status \"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62\": rpc error: code = NotFound desc = could not find container \"6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62\": container with ID starting with 6f4388182cc3cef88408327d46c45986e281fe5e1346c0da8c705cf0903c1a62 not found: ID does not exist"
Sep 05 02:47:52.084113 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.084092    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-gmtqr"]
Sep 05 02:47:52.808267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808233    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:47:52.808556 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808544    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808557    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808566    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="storage-initializer"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808571    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="storage-initializer"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808581    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808587    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808596    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container"
Sep 05 02:47:52.808603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808601    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808612    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="storage-initializer"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808618    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="storage-initializer"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808630    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808636    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808691    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kserve-container"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808698    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808705    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb30d479-577c-497f-a523-533decce2f1a" containerName="kube-rbac-proxy"
Sep 05 02:47:52.808837 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.808710    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97868fb-8330-41d3-8c34-2d451a2134f3" containerName="kserve-container"
Sep 05 02:47:52.813223 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.813207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.815990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.815969    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 05 02:47:52.816135 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.816003    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 05 02:47:52.822047 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.822027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:47:52.883771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.883735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.883877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.883773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.883877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.883808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r44m\" (UniqueName: \"kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.883877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.883826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.883877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.883865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.970305 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.970282    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb30d479-577c-497f-a523-533decce2f1a" path="/var/lib/kubelet/pods/fb30d479-577c-497f-a523-533decce2f1a/volumes"
Sep 05 02:47:52.984501 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.984561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.984561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r44m\" (UniqueName: \"kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.984561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.984710 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.984890 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.984865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.985259 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.985238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.985311 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.985271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.986958 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.986938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:52.992830 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:52.992811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r44m\" (UniqueName: \"kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-fw2jv\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:53.122899 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:53.122820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:53.396321 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:53.396297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:47:53.398411 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:47:53.398389    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981746d5_0514_4853_9d38_34c630c5e473.slice/crio-70abffb4903ad9b35f0d1f970d9b0e86e4f862b35a03fb2083fd87adb9084108 WatchSource:0}: Error finding container 70abffb4903ad9b35f0d1f970d9b0e86e4f862b35a03fb2083fd87adb9084108: Status 404 returned error can't find the container with id 70abffb4903ad9b35f0d1f970d9b0e86e4f862b35a03fb2083fd87adb9084108
Sep 05 02:47:54.068424 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:54.068382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerStarted","Data":"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"}
Sep 05 02:47:54.068825 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:54.068446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerStarted","Data":"70abffb4903ad9b35f0d1f970d9b0e86e4f862b35a03fb2083fd87adb9084108"}
Sep 05 02:47:58.087978 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:58.087943    2560 generic.go:356] "Generic (PLEG): container finished" podID="981746d5-0514-4853-9d38-34c630c5e473" containerID="58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937" exitCode=0
Sep 05 02:47:58.088356 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:58.088027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerDied","Data":"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"}
Sep 05 02:47:59.095034 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:59.094999    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerStarted","Data":"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"}
Sep 05 02:47:59.095387 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:59.095065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerStarted","Data":"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"}
Sep 05 02:47:59.095387 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:59.095258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:47:59.119635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:47:59.119585    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podStartSLOduration=3.090123899 podStartE2EDuration="7.119571239s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.02944734s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:47:59.118089726 +0000 UTC m=+1822.757301547" watchObservedRunningTime="2026-09-05 02:47:59.119571239 +0000 UTC m=+1822.758783038"
Sep 05 02:48:00.099892 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:00.099852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:48:00.101294 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:00.101264    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:01.104413 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:01.104370    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:01.950145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:01.950103    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:48:06.109543 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:06.109510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:48:06.110136 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:06.110110    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:11.950687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:11.950648    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:48:16.110216 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:16.110175    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:21.950301 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:21.950261    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:48:26.110944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:26.110898    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:31.950270 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:31.950231    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:48:35.965849 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:35.965817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:48:36.110692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:36.110653    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:40.611209 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:40.611177    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:48:40.611586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:40.611412    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container" containerID="cri-o://fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059" gracePeriod=30
Sep 05 02:48:40.611586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:40.611431    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kube-rbac-proxy" containerID="cri-o://f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c" gracePeriod=30
Sep 05 02:48:41.294027 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:41.293996    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerID="f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c" exitCode=2
Sep 05 02:48:41.294199 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:41.294041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerDied","Data":"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"}
Sep 05 02:48:41.944256 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:41.944207    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 05 02:48:44.858948 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.858923    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:48:44.905043 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905014    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location\") pod \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") "
Sep 05 02:48:44.905186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config\") pod \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") "
Sep 05 02:48:44.905186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905089    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert\") pod \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") "
Sep 05 02:48:44.905186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fllm\" (UniqueName: \"kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm\") pod \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") "
Sep 05 02:48:44.905359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905239    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls\") pod \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\" (UID: \"7a9d8777-8b57-45aa-bc2e-266713ca68e9\") "
Sep 05 02:48:44.905426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a9d8777-8b57-45aa-bc2e-266713ca68e9" (UID: "7a9d8777-8b57-45aa-bc2e-266713ca68e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:44.905486 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905443    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a9d8777-8b57-45aa-bc2e-266713ca68e9" (UID: "7a9d8777-8b57-45aa-bc2e-266713ca68e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:48:44.905540 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a9d8777-8b57-45aa-bc2e-266713ca68e9" (UID: "7a9d8777-8b57-45aa-bc2e-266713ca68e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:48:44.905540 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905514    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:48:44.905540 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.905527    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:48:44.907260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.907235    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm" (OuterVolumeSpecName: "kube-api-access-2fllm") pod "7a9d8777-8b57-45aa-bc2e-266713ca68e9" (UID: "7a9d8777-8b57-45aa-bc2e-266713ca68e9"). InnerVolumeSpecName "kube-api-access-2fllm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:44.907328 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:44.907254    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a9d8777-8b57-45aa-bc2e-266713ca68e9" (UID: "7a9d8777-8b57-45aa-bc2e-266713ca68e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:45.006015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.005991    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a9d8777-8b57-45aa-bc2e-266713ca68e9-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:48:45.006015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.006011    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a9d8777-8b57-45aa-bc2e-266713ca68e9-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:48:45.006138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.006021    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fllm\" (UniqueName: \"kubernetes.io/projected/7a9d8777-8b57-45aa-bc2e-266713ca68e9-kube-api-access-2fllm\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:48:45.319832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.319727    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerID="fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059" exitCode=0
Sep 05 02:48:45.319832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.319810    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"
Sep 05 02:48:45.320015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.319800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerDied","Data":"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"}
Sep 05 02:48:45.320015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.319941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx" event={"ID":"7a9d8777-8b57-45aa-bc2e-266713ca68e9","Type":"ContainerDied","Data":"fe393b96cfbdf506e4380541018f9a2e6497a02aaf1214b90e53f1f595995d4b"}
Sep 05 02:48:45.320015 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.319967    2560 scope.go:117] "RemoveContainer" containerID="f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"
Sep 05 02:48:45.328138 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.328116    2560 scope.go:117] "RemoveContainer" containerID="fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"
Sep 05 02:48:45.335561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.335540    2560 scope.go:117] "RemoveContainer" containerID="fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"
Sep 05 02:48:45.338460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.338437    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:48:45.343499 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.343480    2560 scope.go:117] "RemoveContainer" containerID="f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"
Sep 05 02:48:45.343788 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:48:45.343763    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c\": container with ID starting with f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c not found: ID does not exist" containerID="f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"
Sep 05 02:48:45.343879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.343801    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c"} err="failed to get container status \"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c\": rpc error: code = NotFound desc = could not find container \"f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c\": container with ID starting with f7aca6db752063a84e89ad41f7a841e2f11c4baed0267c45e974c4ca97c1fc4c not found: ID does not exist"
Sep 05 02:48:45.343879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.343825    2560 scope.go:117] "RemoveContainer" containerID="fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"
Sep 05 02:48:45.344102 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:48:45.344085    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059\": container with ID starting with fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059 not found: ID does not exist" containerID="fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"
Sep 05 02:48:45.344149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.344107    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059"} err="failed to get container status \"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059\": rpc error: code = NotFound desc = could not find container \"fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059\": container with ID starting with fdd92775ddd515f4ab1b3847753b62d78cbcc3d009d3c645303061e119dbf059 not found: ID does not exist"
Sep 05 02:48:45.344149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.344122    2560 scope.go:117] "RemoveContainer" containerID="fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"
Sep 05 02:48:45.344416 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:48:45.344400    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e\": container with ID starting with fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e not found: ID does not exist" containerID="fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"
Sep 05 02:48:45.344464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.344423    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e"} err="failed to get container status \"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e\": rpc error: code = NotFound desc = could not find container \"fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e\": container with ID starting with fab598b142f4f474c1ac19aa74bbb82bcb41bee0e42ce2e3160dab167c88807e not found: ID does not exist"
Sep 05 02:48:45.344567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.344551    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5cffc756c9-lm9bx"]
Sep 05 02:48:45.800502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800470    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:48:45.800814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800801    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container"
Sep 05 02:48:45.800814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800815    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container"
Sep 05 02:48:45.800903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800823    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kube-rbac-proxy"
Sep 05 02:48:45.800903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800829    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kube-rbac-proxy"
Sep 05 02:48:45.800903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800843    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="storage-initializer"
Sep 05 02:48:45.800903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800849    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="storage-initializer"
Sep 05 02:48:45.800903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800901    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kserve-container"
Sep 05 02:48:45.801066 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.800911    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" containerName="kube-rbac-proxy"
Sep 05 02:48:45.805341 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.805316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:45.807898 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.807880    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 05 02:48:45.808024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.808007    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 05 02:48:45.814091 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.814071    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:48:45.914110 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.914078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:45.914446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.914117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:45.914446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.914178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:45.914446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.914204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlc4\" (UniqueName: \"kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:45.914446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:45.914238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.014756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.014715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.014866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.014798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlc4\" (UniqueName: \"kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.014866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.014856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.014980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.014891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.014980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.014926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.015073 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:48:46.015021    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 05 02:48:46.015124 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:48:46.015107    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls podName:962642d7-0540-41d1-9d4e-19909634704c nodeName:}" failed. No retries permitted until 2026-09-05 02:48:46.515084821 +0000 UTC m=+1870.154296607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" (UID: "962642d7-0540-41d1-9d4e-19909634704c") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 05 02:48:46.015350 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.015323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.015507 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.015489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.015560 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.015543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.023651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.023620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlc4\" (UniqueName: \"kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.110198 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.110134    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:46.519106 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.519076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.521439 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.521411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.715871 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.715840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:46.970637 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.970606    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a9d8777-8b57-45aa-bc2e-266713ca68e9" path="/var/lib/kubelet/pods/7a9d8777-8b57-45aa-bc2e-266713ca68e9/volumes"
Sep 05 02:48:46.990311 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:46.990290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:48:46.992145 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:48:46.992114    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962642d7_0540_41d1_9d4e_19909634704c.slice/crio-9241788fa4f71080829ddc5e559b412f0f00e6148bae38449388de829c52cdcd WatchSource:0}: Error finding container 9241788fa4f71080829ddc5e559b412f0f00e6148bae38449388de829c52cdcd: Status 404 returned error can't find the container with id 9241788fa4f71080829ddc5e559b412f0f00e6148bae38449388de829c52cdcd
Sep 05 02:48:47.331937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:47.331839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerStarted","Data":"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"}
Sep 05 02:48:47.331937 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:47.331904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerStarted","Data":"9241788fa4f71080829ddc5e559b412f0f00e6148bae38449388de829c52cdcd"}
Sep 05 02:48:51.352819 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:51.352786    2560 generic.go:356] "Generic (PLEG): container finished" podID="962642d7-0540-41d1-9d4e-19909634704c" containerID="c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19" exitCode=0
Sep 05 02:48:51.353165 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:51.352842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerDied","Data":"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"}
Sep 05 02:48:52.359633 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:52.359600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerStarted","Data":"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"}
Sep 05 02:48:52.360031 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:52.359668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerStarted","Data":"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"}
Sep 05 02:48:52.360031 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:52.359887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:52.379509 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:52.379464    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podStartSLOduration=3.272873414 podStartE2EDuration="7.379450936s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.106577522s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:48:52.376934844 +0000 UTC m=+1876.016146671" watchObservedRunningTime="2026-09-05 02:48:52.379450936 +0000 UTC m=+1876.018662736"
Sep 05 02:48:53.364529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:53.364496    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:53.365824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:53.365793    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:48:54.368810 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:54.368763    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:48:56.109997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:56.109952    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:48:59.373856 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:59.373779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:48:59.374299 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:48:59.374272    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:06.110663 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:06.110622    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:49:09.375190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:09.375151    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:12.971359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:12.971327    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:49:19.374830 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:19.374788    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:22.759137 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.759106    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:49:22.759562 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.759360    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" containerID="cri-o://b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244" gracePeriod=30
Sep 05 02:49:22.760281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.760236    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kube-rbac-proxy" containerID="cri-o://b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81" gracePeriod=30
Sep 05 02:49:22.857491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.857462    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:49:22.861149 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.861130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.863678 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.863656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 05 02:49:22.874228 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.874205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:49:22.911197 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.911177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.911297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.911215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfr4\" (UniqueName: \"kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.911297 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.911248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.911407 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.911326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.911453 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.911413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:22.966850 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:22.966814    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:49:23.012373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.012373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfr4\" (UniqueName: \"kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.012491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.012491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.012491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.012834 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.012819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.013017 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.013000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.013093 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.013076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.014712 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.014694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.021246 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.021226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfr4\" (UniqueName: \"kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-z6pdz\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.170873 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.170839    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:23.441882 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.441852    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:49:23.443654 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:49:23.443629    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19dac07_ca7e_42e7_9f1c_6a06978e1a93.slice/crio-6c596a182a94c9704c265e7178db960d44a9c4e6ed5c34eb2ec8fa13a1f65133 WatchSource:0}: Error finding container 6c596a182a94c9704c265e7178db960d44a9c4e6ed5c34eb2ec8fa13a1f65133: Status 404 returned error can't find the container with id 6c596a182a94c9704c265e7178db960d44a9c4e6ed5c34eb2ec8fa13a1f65133
Sep 05 02:49:23.504602 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.504574    2560 generic.go:356] "Generic (PLEG): container finished" podID="981746d5-0514-4853-9d38-34c630c5e473" containerID="b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81" exitCode=2
Sep 05 02:49:23.504695 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.504646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerDied","Data":"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"}
Sep 05 02:49:23.506290 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.506269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerStarted","Data":"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"}
Sep 05 02:49:23.506395 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:23.506324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerStarted","Data":"6c596a182a94c9704c265e7178db960d44a9c4e6ed5c34eb2ec8fa13a1f65133"}
Sep 05 02:49:26.104807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:26.104769    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 05 02:49:27.109965 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.109945    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:49:27.144067 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls\") pod \"981746d5-0514-4853-9d38-34c630c5e473\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") "
Sep 05 02:49:27.144173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r44m\" (UniqueName: \"kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m\") pod \"981746d5-0514-4853-9d38-34c630c5e473\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") "
Sep 05 02:49:27.144173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location\") pod \"981746d5-0514-4853-9d38-34c630c5e473\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") "
Sep 05 02:49:27.144280 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config\") pod \"981746d5-0514-4853-9d38-34c630c5e473\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") "
Sep 05 02:49:27.144280 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert\") pod \"981746d5-0514-4853-9d38-34c630c5e473\" (UID: \"981746d5-0514-4853-9d38-34c630c5e473\") "
Sep 05 02:49:27.144495 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "981746d5-0514-4853-9d38-34c630c5e473" (UID: "981746d5-0514-4853-9d38-34c630c5e473"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:49:27.144572 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144528    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "981746d5-0514-4853-9d38-34c630c5e473" (UID: "981746d5-0514-4853-9d38-34c630c5e473"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:49:27.144572 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144563    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981746d5-0514-4853-9d38-34c630c5e473-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:49:27.144650 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.144597    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "981746d5-0514-4853-9d38-34c630c5e473" (UID: "981746d5-0514-4853-9d38-34c630c5e473"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:49:27.146132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.146109    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "981746d5-0514-4853-9d38-34c630c5e473" (UID: "981746d5-0514-4853-9d38-34c630c5e473"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:49:27.146213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.146170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m" (OuterVolumeSpecName: "kube-api-access-4r44m") pod "981746d5-0514-4853-9d38-34c630c5e473" (UID: "981746d5-0514-4853-9d38-34c630c5e473"). InnerVolumeSpecName "kube-api-access-4r44m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:49:27.245484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.245422    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981746d5-0514-4853-9d38-34c630c5e473-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:49:27.245484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.245446    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r44m\" (UniqueName: \"kubernetes.io/projected/981746d5-0514-4853-9d38-34c630c5e473-kube-api-access-4r44m\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:49:27.245484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.245455    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:49:27.245484 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.245465    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981746d5-0514-4853-9d38-34c630c5e473-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:49:27.526081 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.525999    2560 generic.go:356] "Generic (PLEG): container finished" podID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerID="560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b" exitCode=0
Sep 05 02:49:27.526239 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.526076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerDied","Data":"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"}
Sep 05 02:49:27.528096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.528068    2560 generic.go:356] "Generic (PLEG): container finished" podID="981746d5-0514-4853-9d38-34c630c5e473" containerID="b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244" exitCode=0
Sep 05 02:49:27.528213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.528136    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"
Sep 05 02:49:27.528213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.528149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerDied","Data":"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"}
Sep 05 02:49:27.528213 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.528206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv" event={"ID":"981746d5-0514-4853-9d38-34c630c5e473","Type":"ContainerDied","Data":"70abffb4903ad9b35f0d1f970d9b0e86e4f862b35a03fb2083fd87adb9084108"}
Sep 05 02:49:27.528323 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.528233    2560 scope.go:117] "RemoveContainer" containerID="b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"
Sep 05 02:49:27.540101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.540083    2560 scope.go:117] "RemoveContainer" containerID="b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"
Sep 05 02:49:27.548680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.548662    2560 scope.go:117] "RemoveContainer" containerID="58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"
Sep 05 02:49:27.559023 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.559006    2560 scope.go:117] "RemoveContainer" containerID="b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"
Sep 05 02:49:27.559338 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:49:27.559315    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81\": container with ID starting with b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81 not found: ID does not exist" containerID="b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"
Sep 05 02:49:27.559427 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.559342    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81"} err="failed to get container status \"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81\": rpc error: code = NotFound desc = could not find container \"b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81\": container with ID starting with b8470565b3cd8ef83bd19fcb644051b8ed6378c45f482ce15817f149c73c3a81 not found: ID does not exist"
Sep 05 02:49:27.559427 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.559359    2560 scope.go:117] "RemoveContainer" containerID="b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"
Sep 05 02:49:27.559649 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:49:27.559630    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244\": container with ID starting with b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244 not found: ID does not exist" containerID="b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"
Sep 05 02:49:27.559724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.559657    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244"} err="failed to get container status \"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244\": rpc error: code = NotFound desc = could not find container \"b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244\": container with ID starting with b1dbc06aacd1ff4f1230475a8d6bb8c4fa07b2ccfd753c6959bd69010c5b2244 not found: ID does not exist"
Sep 05 02:49:27.559724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.559680    2560 scope.go:117] "RemoveContainer" containerID="58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"
Sep 05 02:49:27.559998 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:49:27.559975    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937\": container with ID starting with 58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937 not found: ID does not exist" containerID="58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"
Sep 05 02:49:27.560049 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.560005    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937"} err="failed to get container status \"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937\": rpc error: code = NotFound desc = could not find container \"58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937\": container with ID starting with 58a5d1727d6573270e06ffc15bdec967eac7c25f8c8488dd440c5b4cce4bc937 not found: ID does not exist"
Sep 05 02:49:27.564054 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.564035    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:49:27.565726 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:27.565709    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-fw2jv"]
Sep 05 02:49:28.535921 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.535881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerStarted","Data":"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"}
Sep 05 02:49:28.536404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.535967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerStarted","Data":"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"}
Sep 05 02:49:28.536491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.536416    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:28.536624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.536605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:28.537988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.537953    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:28.556866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.556822    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podStartSLOduration=2.7319425429999997 podStartE2EDuration="6.556808243s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8248657s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:49:28.55531358 +0000 UTC m=+1912.194525397" watchObservedRunningTime="2026-09-05 02:49:28.556808243 +0000 UTC m=+1912.196020142"
Sep 05 02:49:28.970190 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:28.970155    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981746d5-0514-4853-9d38-34c630c5e473" path="/var/lib/kubelet/pods/981746d5-0514-4853-9d38-34c630c5e473/volumes"
Sep 05 02:49:29.374445 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:29.374365    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:29.542657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:29.542615    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:30.546639 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:30.546600    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:35.551716 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:35.551667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:49:35.552205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:35.552178    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:39.375132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:39.375092    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:45.552147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:45.552108    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:49.374641 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:49.374597    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:55.552570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:55.552529    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:49:59.374505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:59.374465    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:49:59.965784 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:49:59.965726    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:50:05.552690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:05.552645    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:50:09.966888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:09.966861    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:50:15.552874 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:15.552825    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:50:15.949319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:15.949284    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:50:15.949582 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:15.949557    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" containerID="cri-o://a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0" gracePeriod=30
Sep 05 02:50:15.949666 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:15.949636    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kube-rbac-proxy" containerID="cri-o://892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51" gracePeriod=30
Sep 05 02:50:16.759403 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:16.759370    2560 generic.go:356] "Generic (PLEG): container finished" podID="962642d7-0540-41d1-9d4e-19909634704c" containerID="892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51" exitCode=2
Sep 05 02:50:16.759787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:16.759441    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerDied","Data":"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"}
Sep 05 02:50:19.369889 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:19.369849    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 05 02:50:19.965679 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:19.965642    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 05 02:50:20.492433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.492409    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:50:20.670107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vlc4\" (UniqueName: \"kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4\") pod \"962642d7-0540-41d1-9d4e-19909634704c\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") "
Sep 05 02:50:20.670264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location\") pod \"962642d7-0540-41d1-9d4e-19909634704c\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") "
Sep 05 02:50:20.670264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670184    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") pod \"962642d7-0540-41d1-9d4e-19909634704c\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") "
Sep 05 02:50:20.670264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert\") pod \"962642d7-0540-41d1-9d4e-19909634704c\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") "
Sep 05 02:50:20.670264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config\") pod \"962642d7-0540-41d1-9d4e-19909634704c\" (UID: \"962642d7-0540-41d1-9d4e-19909634704c\") "
Sep 05 02:50:20.670491 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "962642d7-0540-41d1-9d4e-19909634704c" (UID: "962642d7-0540-41d1-9d4e-19909634704c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:20.670641 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "962642d7-0540-41d1-9d4e-19909634704c" (UID: "962642d7-0540-41d1-9d4e-19909634704c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:20.670705 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.670666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "962642d7-0540-41d1-9d4e-19909634704c" (UID: "962642d7-0540-41d1-9d4e-19909634704c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:20.672220 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.672201    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "962642d7-0540-41d1-9d4e-19909634704c" (UID: "962642d7-0540-41d1-9d4e-19909634704c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:20.672303 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.672278    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4" (OuterVolumeSpecName: "kube-api-access-8vlc4") pod "962642d7-0540-41d1-9d4e-19909634704c" (UID: "962642d7-0540-41d1-9d4e-19909634704c"). InnerVolumeSpecName "kube-api-access-8vlc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:20.771362 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.771326    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:20.771362 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.771359    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/962642d7-0540-41d1-9d4e-19909634704c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:20.771362 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.771369    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlc4\" (UniqueName: \"kubernetes.io/projected/962642d7-0540-41d1-9d4e-19909634704c-kube-api-access-8vlc4\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:20.771545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.771378    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/962642d7-0540-41d1-9d4e-19909634704c-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:20.771545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.771387    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/962642d7-0540-41d1-9d4e-19909634704c-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:20.779177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.779147    2560 generic.go:356] "Generic (PLEG): container finished" podID="962642d7-0540-41d1-9d4e-19909634704c" containerID="a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0" exitCode=0
Sep 05 02:50:20.779250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.779214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"
Sep 05 02:50:20.779308 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.779212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerDied","Data":"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"}
Sep 05 02:50:20.779386 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.779368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx" event={"ID":"962642d7-0540-41d1-9d4e-19909634704c","Type":"ContainerDied","Data":"9241788fa4f71080829ddc5e559b412f0f00e6148bae38449388de829c52cdcd"}
Sep 05 02:50:20.779446 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.779422    2560 scope.go:117] "RemoveContainer" containerID="892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"
Sep 05 02:50:20.788429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.788409    2560 scope.go:117] "RemoveContainer" containerID="a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"
Sep 05 02:50:20.795722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.795703    2560 scope.go:117] "RemoveContainer" containerID="c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"
Sep 05 02:50:20.802774 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.802751    2560 scope.go:117] "RemoveContainer" containerID="892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"
Sep 05 02:50:20.803040 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:20.803021    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51\": container with ID starting with 892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51 not found: ID does not exist" containerID="892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"
Sep 05 02:50:20.803105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.803049    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51"} err="failed to get container status \"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51\": rpc error: code = NotFound desc = could not find container \"892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51\": container with ID starting with 892b0726e27bbc5efb7580f224a9cacd742d5a528f315fd0a81e6245388f3a51 not found: ID does not exist"
Sep 05 02:50:20.803105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.803068    2560 scope.go:117] "RemoveContainer" containerID="a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"
Sep 05 02:50:20.803308 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:20.803291    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0\": container with ID starting with a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0 not found: ID does not exist" containerID="a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"
Sep 05 02:50:20.803351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.803314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0"} err="failed to get container status \"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0\": rpc error: code = NotFound desc = could not find container \"a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0\": container with ID starting with a1b705414a6d904f4a4d2c76d78de059054596356519e6b74691b6da9b8f21d0 not found: ID does not exist"
Sep 05 02:50:20.803351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.803329    2560 scope.go:117] "RemoveContainer" containerID="c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"
Sep 05 02:50:20.803559 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:20.803544    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19\": container with ID starting with c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19 not found: ID does not exist" containerID="c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"
Sep 05 02:50:20.803599 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.803563    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19"} err="failed to get container status \"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19\": rpc error: code = NotFound desc = could not find container \"c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19\": container with ID starting with c7944d34f00f1cda6857319141e1dc7828752bf565750770b726961654697b19 not found: ID does not exist"
Sep 05 02:50:20.814181 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.814160    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:50:20.817332 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.817313    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-546d4d597d-lkqwx"]
Sep 05 02:50:20.969588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:20.969528    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="962642d7-0540-41d1-9d4e-19909634704c" path="/var/lib/kubelet/pods/962642d7-0540-41d1-9d4e-19909634704c/volumes"
Sep 05 02:50:25.552942 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:25.552903    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:50:35.552965 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:35.552878    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:50:45.552905 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:45.552875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:50:52.958405 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:52.958365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:50:52.958889 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:52.958623    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" containerID="cri-o://361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368" gracePeriod=30
Sep 05 02:50:52.958889 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:52.958665    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kube-rbac-proxy" containerID="cri-o://9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1" gracePeriod=30
Sep 05 02:50:53.933415 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:53.933380    2560 generic.go:356] "Generic (PLEG): container finished" podID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerID="9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1" exitCode=2
Sep 05 02:50:53.933572 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:53.933449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerDied","Data":"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"}
Sep 05 02:50:55.547458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:55.547414    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 05 02:50:55.552690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:55.552664    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 05 02:50:57.196810 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.196788    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:50:57.253275 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert\") pod \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") "
Sep 05 02:50:57.253421 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location\") pod \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") "
Sep 05 02:50:57.253421 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253358    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhfr4\" (UniqueName: \"kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4\") pod \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") "
Sep 05 02:50:57.253421 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls\") pod \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") "
Sep 05 02:50:57.253421 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config\") pod \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\" (UID: \"e19dac07-ca7e-42e7-9f1c-6a06978e1a93\") "
Sep 05 02:50:57.253713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e19dac07-ca7e-42e7-9f1c-6a06978e1a93" (UID: "e19dac07-ca7e-42e7-9f1c-6a06978e1a93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:57.253713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e19dac07-ca7e-42e7-9f1c-6a06978e1a93" (UID: "e19dac07-ca7e-42e7-9f1c-6a06978e1a93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:57.253903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.253774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e19dac07-ca7e-42e7-9f1c-6a06978e1a93" (UID: "e19dac07-ca7e-42e7-9f1c-6a06978e1a93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:57.255433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.255406    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4" (OuterVolumeSpecName: "kube-api-access-jhfr4") pod "e19dac07-ca7e-42e7-9f1c-6a06978e1a93" (UID: "e19dac07-ca7e-42e7-9f1c-6a06978e1a93"). InnerVolumeSpecName "kube-api-access-jhfr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:57.255433 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.255418    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e19dac07-ca7e-42e7-9f1c-6a06978e1a93" (UID: "e19dac07-ca7e-42e7-9f1c-6a06978e1a93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:57.354062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.354009    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:57.354062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.354032    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:57.354062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.354041    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfr4\" (UniqueName: \"kubernetes.io/projected/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-api-access-jhfr4\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:57.354062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.354050    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:57.354062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.354061    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19dac07-ca7e-42e7-9f1c-6a06978e1a93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:50:57.953781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.953726    2560 generic.go:356] "Generic (PLEG): container finished" podID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerID="361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368" exitCode=0
Sep 05 02:50:57.953943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.953792    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"
Sep 05 02:50:57.953943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.953815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerDied","Data":"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"}
Sep 05 02:50:57.953943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.953873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz" event={"ID":"e19dac07-ca7e-42e7-9f1c-6a06978e1a93","Type":"ContainerDied","Data":"6c596a182a94c9704c265e7178db960d44a9c4e6ed5c34eb2ec8fa13a1f65133"}
Sep 05 02:50:57.953943 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.953900    2560 scope.go:117] "RemoveContainer" containerID="9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"
Sep 05 02:50:57.962657 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.962640    2560 scope.go:117] "RemoveContainer" containerID="361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"
Sep 05 02:50:57.969709 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.969690    2560 scope.go:117] "RemoveContainer" containerID="560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"
Sep 05 02:50:57.977273 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977256    2560 scope.go:117] "RemoveContainer" containerID="9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"
Sep 05 02:50:57.977514 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:57.977499    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1\": container with ID starting with 9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1 not found: ID does not exist" containerID="9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"
Sep 05 02:50:57.977561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977521    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1"} err="failed to get container status \"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1\": rpc error: code = NotFound desc = could not find container \"9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1\": container with ID starting with 9843b63db3391ba9f5e45aafe45bcdf462ad7c3c16ce5f56b880fe20e46545b1 not found: ID does not exist"
Sep 05 02:50:57.977561 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977536    2560 scope.go:117] "RemoveContainer" containerID="361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"
Sep 05 02:50:57.977676 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977572    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:50:57.977830 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:57.977814    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368\": container with ID starting with 361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368 not found: ID does not exist" containerID="361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"
Sep 05 02:50:57.977881 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977833    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368"} err="failed to get container status \"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368\": rpc error: code = NotFound desc = could not find container \"361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368\": container with ID starting with 361f3021c45174ffd1392b72fc698c1438751ab845eb61fe57ee8239f8dbc368 not found: ID does not exist"
Sep 05 02:50:57.977881 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.977849    2560 scope.go:117] "RemoveContainer" containerID="560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"
Sep 05 02:50:57.978067 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:57.978053    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b\": container with ID starting with 560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b not found: ID does not exist" containerID="560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"
Sep 05 02:50:57.978105 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.978069    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b"} err="failed to get container status \"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b\": rpc error: code = NotFound desc = could not find container \"560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b\": container with ID starting with 560e25bd3899b1d6b1b7a8367fb94afb2a5df21a10f10258731ee91ab616872b not found: ID does not exist"
Sep 05 02:50:57.980537 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:57.980517    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-z6pdz"]
Sep 05 02:50:58.197586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.197548    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:50:58.198052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198012    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container"
Sep 05 02:50:58.198052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198032    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198053    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198062    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198072    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198081    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198095    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="storage-initializer"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198104    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="storage-initializer"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198114    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198134    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="storage-initializer"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198143    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="storage-initializer"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198153    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198162    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container"
Sep 05 02:50:58.198175 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198175    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="storage-initializer"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198183    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="storage-initializer"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198196    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198205    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198286    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198299    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="962642d7-0540-41d1-9d4e-19909634704c" containerName="kserve-container"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198307    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kserve-container"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198316    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" containerName="kube-rbac-proxy"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198329    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kserve-container"
Sep 05 02:50:58.198888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.198337    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="981746d5-0514-4853-9d38-34c630c5e473" containerName="kube-rbac-proxy"
Sep 05 02:50:58.203264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.203245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.205861 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.205798    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:50:58.206013 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.205996    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 05 02:50:58.206281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.206262    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:50:58.206418 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.206303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:50:58.206418 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.206262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 05 02:50:58.206418 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.206264    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:50:58.206585 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.206418    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:50:58.211489 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.211469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:50:58.262111 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.262083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcdt\" (UniqueName: \"kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.262220 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.262129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.262220 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.262186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.262306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.262246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.262306 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.262289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.362954 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.362930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcdt\" (UniqueName: \"kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363060 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.362963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363060 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.362994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363202 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:58.363179    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 05 02:50:58.363202 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.363195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363394 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:50:58.363248    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls podName:794278dc-91ba-4e39-8b3c-b157aff68337 nodeName:}" failed. No retries permitted until 2026-09-05 02:50:58.863227912 +0000 UTC m=+2002.502439696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls") pod "isvc-canary-rollback-predictor-76998db6fd-5s726" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 05 02:50:58.363394 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.363270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363659 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.363636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.363658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.363864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.363846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.371799 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.371780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcdt\" (UniqueName: \"kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.867286 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.867254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.869621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.869588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-5s726\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:58.969483 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:58.969446    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19dac07-ca7e-42e7-9f1c-6a06978e1a93" path="/var/lib/kubelet/pods/e19dac07-ca7e-42e7-9f1c-6a06978e1a93/volumes"
Sep 05 02:50:59.115241 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:59.115219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:50:59.389322 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:59.389293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:50:59.392456 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:50:59.392432    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794278dc_91ba_4e39_8b3c_b157aff68337.slice/crio-ae6504a8fe931f186ed7953903d0bae95df4e513577cf4cf817c25cef62b8cb7 WatchSource:0}: Error finding container ae6504a8fe931f186ed7953903d0bae95df4e513577cf4cf817c25cef62b8cb7: Status 404 returned error can't find the container with id ae6504a8fe931f186ed7953903d0bae95df4e513577cf4cf817c25cef62b8cb7
Sep 05 02:50:59.964690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:59.964653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerStarted","Data":"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"}
Sep 05 02:50:59.964842 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:50:59.964712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerStarted","Data":"ae6504a8fe931f186ed7953903d0bae95df4e513577cf4cf817c25cef62b8cb7"}
Sep 05 02:51:03.983972 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:03.983933    2560 generic.go:356] "Generic (PLEG): container finished" podID="794278dc-91ba-4e39-8b3c-b157aff68337" containerID="c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa" exitCode=0
Sep 05 02:51:03.984389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:03.984010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerDied","Data":"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"}
Sep 05 02:51:04.990395 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:04.990360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerStarted","Data":"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"}
Sep 05 02:51:04.990789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:04.990416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerStarted","Data":"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"}
Sep 05 02:51:04.990789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:04.990594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:51:05.011859 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:05.011810    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podStartSLOduration=3.36502046 podStartE2EDuration="7.011797226s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.646776766s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:05.010141639 +0000 UTC m=+2008.649353459" watchObservedRunningTime="2026-09-05 02:51:05.011797226 +0000 UTC m=+2008.651009026"
Sep 05 02:51:05.995398 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:05.995366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:51:05.996529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:05.996501    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:06.999468 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:06.999430    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:07.632416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.632377    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:51:07.636084 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.636062    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.638807 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.638781    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:51:07.638939 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.638819    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 05 02:51:07.646786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.646767    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:51:07.739230 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.739207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.739337 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.739237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.739337 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.739261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.739416 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.739389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.739452 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.739436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6tn\" (UniqueName: \"kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.840533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.840505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.840656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.840540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.840656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.840573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.840656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.840611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.840656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.840636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6tn\" (UniqueName: \"kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.841161 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.841135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.841363 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.841343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.841415 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.841363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.843167 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.843145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.849361 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.849338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6tn\" (UniqueName: \"kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:07.946603 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:07.946583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:08.218661 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:08.218632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:51:08.220869 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:51:08.220841    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145c8785_6c22_4146_a66f_7fc28ba5cda4.slice/crio-33471f35b341dbf2d3158167f61329c8f74d4e004e5053eaf7298d47c64e2dc9 WatchSource:0}: Error finding container 33471f35b341dbf2d3158167f61329c8f74d4e004e5053eaf7298d47c64e2dc9: Status 404 returned error can't find the container with id 33471f35b341dbf2d3158167f61329c8f74d4e004e5053eaf7298d47c64e2dc9
Sep 05 02:51:09.013626 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:09.013590    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerStarted","Data":"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"}
Sep 05 02:51:09.013866 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:09.013647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerStarted","Data":"33471f35b341dbf2d3158167f61329c8f74d4e004e5053eaf7298d47c64e2dc9"}
Sep 05 02:51:12.004846 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:12.004817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:51:12.005357 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:12.005325    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:13.035280 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:13.035245    2560 generic.go:356] "Generic (PLEG): container finished" podID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerID="3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809" exitCode=0
Sep 05 02:51:13.035629 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:13.035334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerDied","Data":"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"}
Sep 05 02:51:13.036713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:13.036697    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:51:17.060417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.060332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerStarted","Data":"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"}
Sep 05 02:51:17.060417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.060410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerStarted","Data":"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"}
Sep 05 02:51:17.060901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.060662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:17.060901 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.060823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:17.062061 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.062030    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 05 02:51:17.080713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:17.080674    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podStartSLOduration=1.907740375 podStartE2EDuration="10.080662285s" totalImagesPullingTime="3.628618848s" totalInitContainerRuntime="4.544303062s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:17.07982837 +0000 UTC m=+2020.719040199" watchObservedRunningTime="2026-09-05 02:51:17.080662285 +0000 UTC m=+2020.719874084"
Sep 05 02:51:18.065078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:18.065034    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 05 02:51:22.006057 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:22.006012    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:23.069589 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:23.069561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:23.070186 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:23.070157    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 05 02:51:32.006183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:32.006145    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:33.071021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:33.070986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:42.005853 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:42.005811    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:47.367480 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:47.367443    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:51:47.367865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:47.367686    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container" containerID="cri-o://32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e" gracePeriod=30
Sep 05 02:51:47.367865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:47.367761    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" containerID="cri-o://2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9" gracePeriod=30
Sep 05 02:51:48.065406 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:48.065366    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:51:48.209395 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:48.209361    2560 generic.go:356] "Generic (PLEG): container finished" podID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerID="2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9" exitCode=2
Sep 05 02:51:48.209554 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:48.209433    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerDied","Data":"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"}
Sep 05 02:51:52.005474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.005436    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:51:52.536305 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.536272    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:51:52.539898 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.539881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.542392 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.542368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 05 02:51:52.542392 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.542388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 05 02:51:52.550000 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.549980    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:51:52.594502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.594479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckbg\" (UniqueName: \"kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.594611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.594512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.594611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.594540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.594727 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.594643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.594727 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.594681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695101 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695210 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695311 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckbg\" (UniqueName: \"kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695365 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695497 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.695573 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:51:52.695485    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 05 02:51:52.695573 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:51:52.695559    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls podName:eeecadba-c70b-4516-ba4f-393220473c7b nodeName:}" failed. No retries permitted until 2026-09-05 02:51:53.195538761 +0000 UTC m=+2056.834750544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls") pod "isvc-triton-predictor-5d7db58676-7ftws" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b") : secret "isvc-triton-predictor-serving-cert" not found
Sep 05 02:51:52.696022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.695994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.696411 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.696388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:52.704143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:52.704114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckbg\" (UniqueName: \"kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:53.065920 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:53.065878    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:51:53.200478 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:53.200443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:53.200671 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:51:53.200591    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 05 02:51:53.200671 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:51:53.200654    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls podName:eeecadba-c70b-4516-ba4f-393220473c7b nodeName:}" failed. No retries permitted until 2026-09-05 02:51:54.200640085 +0000 UTC m=+2057.839851868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls") pod "isvc-triton-predictor-5d7db58676-7ftws" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b") : secret "isvc-triton-predictor-serving-cert" not found
Sep 05 02:51:54.209544 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:54.209512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:54.211988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:54.211962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7ftws\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:54.351762 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:54.351723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:51:54.623403 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:54.623356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:51:54.625889 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:51:54.625859    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeecadba_c70b_4516_ba4f_393220473c7b.slice/crio-406ab87eeb1e44602b5f0f098446af38de7b673d825f16efc01fd5cbe73f79e9 WatchSource:0}: Error finding container 406ab87eeb1e44602b5f0f098446af38de7b673d825f16efc01fd5cbe73f79e9: Status 404 returned error can't find the container with id 406ab87eeb1e44602b5f0f098446af38de7b673d825f16efc01fd5cbe73f79e9
Sep 05 02:51:55.249526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:55.249496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerStarted","Data":"66a77003f698d07e67d257068b5a00f3aa557d9478488f0135199ada9579763d"}
Sep 05 02:51:55.250004 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:55.249567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerStarted","Data":"406ab87eeb1e44602b5f0f098446af38de7b673d825f16efc01fd5cbe73f79e9"}
Sep 05 02:51:58.066244 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:58.066202    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:51:58.066621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:58.066314    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:51:59.270646 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:59.270603    2560 generic.go:356] "Generic (PLEG): container finished" podID="eeecadba-c70b-4516-ba4f-393220473c7b" containerID="66a77003f698d07e67d257068b5a00f3aa557d9478488f0135199ada9579763d" exitCode=0
Sep 05 02:51:59.271097 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:51:59.270812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerDied","Data":"66a77003f698d07e67d257068b5a00f3aa557d9478488f0135199ada9579763d"}
Sep 05 02:52:02.005952 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:02.005543    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:52:03.070430 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:03.069690    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:52:08.066315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:08.065835    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:52:12.005689 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:12.005270    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:52:13.066143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:13.065999    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 05 02:52:18.086147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.086116    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:52:18.239445 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.239348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location\") pod \"145c8785-6c22-4146-a66f-7fc28ba5cda4\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") "
Sep 05 02:52:18.239621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.239457    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert\") pod \"145c8785-6c22-4146-a66f-7fc28ba5cda4\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") "
Sep 05 02:52:18.239621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.239486    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn6tn\" (UniqueName: \"kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn\") pod \"145c8785-6c22-4146-a66f-7fc28ba5cda4\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") "
Sep 05 02:52:18.239621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.239544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config\") pod \"145c8785-6c22-4146-a66f-7fc28ba5cda4\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") "
Sep 05 02:52:18.239621 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.239588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls\") pod \"145c8785-6c22-4146-a66f-7fc28ba5cda4\" (UID: \"145c8785-6c22-4146-a66f-7fc28ba5cda4\") "
Sep 05 02:52:18.240949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.240900    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "145c8785-6c22-4146-a66f-7fc28ba5cda4" (UID: "145c8785-6c22-4146-a66f-7fc28ba5cda4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:18.240949 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.240898    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "145c8785-6c22-4146-a66f-7fc28ba5cda4" (UID: "145c8785-6c22-4146-a66f-7fc28ba5cda4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:18.243380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.243351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "145c8785-6c22-4146-a66f-7fc28ba5cda4" (UID: "145c8785-6c22-4146-a66f-7fc28ba5cda4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:52:18.243588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.243561    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn" (OuterVolumeSpecName: "kube-api-access-dn6tn") pod "145c8785-6c22-4146-a66f-7fc28ba5cda4" (UID: "145c8785-6c22-4146-a66f-7fc28ba5cda4"). InnerVolumeSpecName "kube-api-access-dn6tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:52:18.254843 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.254815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "145c8785-6c22-4146-a66f-7fc28ba5cda4" (UID: "145c8785-6c22-4146-a66f-7fc28ba5cda4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:52:18.340722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.340657    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/145c8785-6c22-4146-a66f-7fc28ba5cda4-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:18.340722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.340687    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/145c8785-6c22-4146-a66f-7fc28ba5cda4-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:18.340722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.340702    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:18.340722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.340717    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6tn\" (UniqueName: \"kubernetes.io/projected/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-api-access-dn6tn\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:18.341035 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.340732    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/145c8785-6c22-4146-a66f-7fc28ba5cda4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:18.424622 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.424562    2560 generic.go:356] "Generic (PLEG): container finished" podID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerID="32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e" exitCode=137
Sep 05 02:52:18.424797 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.424633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"
Sep 05 02:52:18.424797 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.424626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerDied","Data":"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"}
Sep 05 02:52:18.424928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.424815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm" event={"ID":"145c8785-6c22-4146-a66f-7fc28ba5cda4","Type":"ContainerDied","Data":"33471f35b341dbf2d3158167f61329c8f74d4e004e5053eaf7298d47c64e2dc9"}
Sep 05 02:52:18.424928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.424861    2560 scope.go:117] "RemoveContainer" containerID="2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"
Sep 05 02:52:18.437771 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.437727    2560 scope.go:117] "RemoveContainer" containerID="32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"
Sep 05 02:52:18.450487 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.450466    2560 scope.go:117] "RemoveContainer" containerID="3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"
Sep 05 02:52:18.450915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.450807    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:52:18.454397 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.454374    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hxrmm"]
Sep 05 02:52:18.461860 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.461840    2560 scope.go:117] "RemoveContainer" containerID="2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"
Sep 05 02:52:18.462255 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:18.462128    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9\": container with ID starting with 2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9 not found: ID does not exist" containerID="2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"
Sep 05 02:52:18.462255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.462163    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9"} err="failed to get container status \"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9\": rpc error: code = NotFound desc = could not find container \"2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9\": container with ID starting with 2dc7385e43340c7a2792332125387ff1269ee7ef415865e7f33ecc6bc29b5cd9 not found: ID does not exist"
Sep 05 02:52:18.462255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.462185    2560 scope.go:117] "RemoveContainer" containerID="32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"
Sep 05 02:52:18.462633 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:18.462515    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e\": container with ID starting with 32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e not found: ID does not exist" containerID="32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"
Sep 05 02:52:18.462633 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.462545    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e"} err="failed to get container status \"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e\": rpc error: code = NotFound desc = could not find container \"32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e\": container with ID starting with 32b15f230f6cbedd150f9e296a34169a465293352e7b80afac044248634b4c3e not found: ID does not exist"
Sep 05 02:52:18.462633 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.462567    2560 scope.go:117] "RemoveContainer" containerID="3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"
Sep 05 02:52:18.462995 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:18.462936    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809\": container with ID starting with 3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809 not found: ID does not exist" containerID="3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"
Sep 05 02:52:18.462995 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.462966    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809"} err="failed to get container status \"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809\": rpc error: code = NotFound desc = could not find container \"3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809\": container with ID starting with 3d91a3f2b071d2f0afeec3ec242e1b1431e19d2b8d2a3e1c4c5e00c915d1c809 not found: ID does not exist"
Sep 05 02:52:18.972545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:18.972510    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" path="/var/lib/kubelet/pods/145c8785-6c22-4146-a66f-7fc28ba5cda4/volumes"
Sep 05 02:52:22.006142 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:22.006080    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 05 02:52:27.668642 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:27.668591    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:52:27.669536 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:27.668889    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container" containerID="cri-o://2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5" gracePeriod=30
Sep 05 02:52:27.669536 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:27.669033    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kube-rbac-proxy" containerID="cri-o://3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6" gracePeriod=30
Sep 05 02:52:27.670831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:27.670704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:52:28.247956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.247869    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:52:28.248440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248412    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy"
Sep 05 02:52:28.248440 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248437    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248461    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="storage-initializer"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248469    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="storage-initializer"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248494    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248504    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248586    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kserve-container"
Sep 05 02:52:28.248635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.248603    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="145c8785-6c22-4146-a66f-7fc28ba5cda4" containerName="kube-rbac-proxy"
Sep 05 02:52:28.270903 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.270864    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:52:28.271046 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.270998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.332708 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.332679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.332893 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.332731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.332893 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.332803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggxv\" (UniqueName: \"kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.333060 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.332952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.333060 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.332992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434476 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggxv\" (UniqueName: \"kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.434958 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.434933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.435250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.435224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.435404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.435384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.437383 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.437355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.446693 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.446668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggxv\" (UniqueName: \"kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv\") pod \"isvc-canary-rollback-predictor-76998db6fd-cmmdh\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.487662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.487631    2560 generic.go:356] "Generic (PLEG): container finished" podID="794278dc-91ba-4e39-8b3c-b157aff68337" containerID="3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6" exitCode=2
Sep 05 02:52:28.487800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.487713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerDied","Data":"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"}
Sep 05 02:52:28.586324 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.586253    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:28.916474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:28.916446    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:52:28.920960 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:52:28.920374    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768fc060_e005_4e74_853f_ca17691a93fe.slice/crio-5307a6c3f765cbb9472dc478ef6be0413a0bda11de70937a349b471edd97f57d WatchSource:0}: Error finding container 5307a6c3f765cbb9472dc478ef6be0413a0bda11de70937a349b471edd97f57d: Status 404 returned error can't find the container with id 5307a6c3f765cbb9472dc478ef6be0413a0bda11de70937a349b471edd97f57d
Sep 05 02:52:29.497622 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:29.497504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerStarted","Data":"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"}
Sep 05 02:52:29.497622 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:29.497578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerStarted","Data":"5307a6c3f765cbb9472dc478ef6be0413a0bda11de70937a349b471edd97f57d"}
Sep 05 02:52:32.000132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:32.000053    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 05 02:52:33.291632 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.291528    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:52:33.381391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381083    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert\") pod \"794278dc-91ba-4e39-8b3c-b157aff68337\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") "
Sep 05 02:52:33.381391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbcdt\" (UniqueName: \"kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt\") pod \"794278dc-91ba-4e39-8b3c-b157aff68337\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") "
Sep 05 02:52:33.381391 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381176    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") pod \"794278dc-91ba-4e39-8b3c-b157aff68337\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") "
Sep 05 02:52:33.381662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config\") pod \"794278dc-91ba-4e39-8b3c-b157aff68337\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") "
Sep 05 02:52:33.381662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381511    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location\") pod \"794278dc-91ba-4e39-8b3c-b157aff68337\" (UID: \"794278dc-91ba-4e39-8b3c-b157aff68337\") "
Sep 05 02:52:33.381662 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "794278dc-91ba-4e39-8b3c-b157aff68337" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:33.381990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "794278dc-91ba-4e39-8b3c-b157aff68337" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:52:33.381990 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.381933    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "794278dc-91ba-4e39-8b3c-b157aff68337" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:33.382188 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.382174    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/794278dc-91ba-4e39-8b3c-b157aff68337-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.382247 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.382198    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.382247 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.382213    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/794278dc-91ba-4e39-8b3c-b157aff68337-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.384719 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.384689    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt" (OuterVolumeSpecName: "kube-api-access-bbcdt") pod "794278dc-91ba-4e39-8b3c-b157aff68337" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337"). InnerVolumeSpecName "kube-api-access-bbcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:52:33.385180 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.385137    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "794278dc-91ba-4e39-8b3c-b157aff68337" (UID: "794278dc-91ba-4e39-8b3c-b157aff68337"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:52:33.483429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.483399    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcdt\" (UniqueName: \"kubernetes.io/projected/794278dc-91ba-4e39-8b3c-b157aff68337-kube-api-access-bbcdt\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.483429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.483433    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/794278dc-91ba-4e39-8b3c-b157aff68337-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.525725 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.525635    2560 generic.go:356] "Generic (PLEG): container finished" podID="794278dc-91ba-4e39-8b3c-b157aff68337" containerID="2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5" exitCode=0
Sep 05 02:52:33.525725 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.525702    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"
Sep 05 02:52:33.525928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.525788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerDied","Data":"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"}
Sep 05 02:52:33.525928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.525843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726" event={"ID":"794278dc-91ba-4e39-8b3c-b157aff68337","Type":"ContainerDied","Data":"ae6504a8fe931f186ed7953903d0bae95df4e513577cf4cf817c25cef62b8cb7"}
Sep 05 02:52:33.525928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.525881    2560 scope.go:117] "RemoveContainer" containerID="3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"
Sep 05 02:52:33.529038 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.529009    2560 generic.go:356] "Generic (PLEG): container finished" podID="768fc060-e005-4e74-853f-ca17691a93fe" containerID="7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553" exitCode=0
Sep 05 02:52:33.529146 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.529113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerDied","Data":"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"}
Sep 05 02:52:33.538034 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.537858    2560 scope.go:117] "RemoveContainer" containerID="2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"
Sep 05 02:52:33.549558 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.549528    2560 scope.go:117] "RemoveContainer" containerID="c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"
Sep 05 02:52:33.560271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.560152    2560 scope.go:117] "RemoveContainer" containerID="3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"
Sep 05 02:52:33.560515 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:33.560490    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6\": container with ID starting with 3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6 not found: ID does not exist" containerID="3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"
Sep 05 02:52:33.560594 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.560531    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6"} err="failed to get container status \"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6\": rpc error: code = NotFound desc = could not find container \"3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6\": container with ID starting with 3429ca339f4097332e2b84076119f9cfe40d52c1f1113457c36ff73eafcf89b6 not found: ID does not exist"
Sep 05 02:52:33.560594 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.560555    2560 scope.go:117] "RemoveContainer" containerID="2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"
Sep 05 02:52:33.561131 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:33.560963    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5\": container with ID starting with 2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5 not found: ID does not exist" containerID="2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"
Sep 05 02:52:33.561131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.560992    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5"} err="failed to get container status \"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5\": rpc error: code = NotFound desc = could not find container \"2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5\": container with ID starting with 2f8c5422d1fa295334936d621bad21a2524f865881b5a5484d4469210677fbf5 not found: ID does not exist"
Sep 05 02:52:33.561131 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.561013    2560 scope.go:117] "RemoveContainer" containerID="c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"
Sep 05 02:52:33.561470 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:52:33.561422    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa\": container with ID starting with c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa not found: ID does not exist" containerID="c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"
Sep 05 02:52:33.561470 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.561456    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa"} err="failed to get container status \"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa\": rpc error: code = NotFound desc = could not find container \"c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa\": container with ID starting with c452004fe660298354a129f9ad456e6b9f1de37d7caf38614073aac20f27e1aa not found: ID does not exist"
Sep 05 02:52:33.564798 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.564756    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:52:33.570690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:33.570669    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-5s726"]
Sep 05 02:52:34.542220 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.542170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerStarted","Data":"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"}
Sep 05 02:52:34.543683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.542260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerStarted","Data":"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"}
Sep 05 02:52:34.543683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.542773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:34.543683 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.542850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:34.544085 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.544056    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:52:34.562690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.562492    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podStartSLOduration=2.297248473 podStartE2EDuration="6.562477045s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.265228572s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:52:34.561767751 +0000 UTC m=+2098.200979552" watchObservedRunningTime="2026-09-05 02:52:34.562477045 +0000 UTC m=+2098.201688848"
Sep 05 02:52:34.972520 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:34.972486    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" path="/var/lib/kubelet/pods/794278dc-91ba-4e39-8b3c-b157aff68337/volumes"
Sep 05 02:52:35.552054 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:35.551853    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:52:36.939183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:36.939151    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:52:36.940477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:36.940435    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:52:40.558814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:40.557811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:52:40.560072 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:40.559863    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:52:50.559814 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:52:50.559776    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:00.558999 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:00.558960    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:10.559832 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:10.559788    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:20.559231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:20.559193    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:30.558894 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:30.558805    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:40.559232 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:40.559194    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:50.559349 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:50.559301    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:53:54.989173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:54.989138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerStarted","Data":"863d86ccef6544edf5b7301521f1fd8f790f50dbdc2f8080959448fca7966759"}
Sep 05 02:53:54.989545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:54.989201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:53:54.989545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:54.989223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerStarted","Data":"88e09b7b3c56ffa8dac862d921499bcc4f3369b04760e15df409296b1f89e3f7"}
Sep 05 02:53:54.989545 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:54.989243    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:53:54.990590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:54.990567    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 05 02:53:55.010915 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:55.010868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podStartSLOduration=3.790190131 podStartE2EDuration="2m3.010854355s" totalImagesPullingTime="1m55.246114902s" totalInitContainerRuntime="3.974549322s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:53:55.010317063 +0000 UTC m=+2178.649528884" watchObservedRunningTime="2026-09-05 02:53:55.010854355 +0000 UTC m=+2178.650066155"
Sep 05 02:53:55.994405 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:55.994366    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 05 02:53:56.967364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:53:56.967329    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 05 02:54:01.009236 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:01.009205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:54:01.010058 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:01.010038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:54:04.191933 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:04.191905    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:54:04.192422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:04.192129    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container" containerID="cri-o://88e09b7b3c56ffa8dac862d921499bcc4f3369b04760e15df409296b1f89e3f7" gracePeriod=30
Sep 05 02:54:04.192422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:04.192167    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kube-rbac-proxy" containerID="cri-o://863d86ccef6544edf5b7301521f1fd8f790f50dbdc2f8080959448fca7966759" gracePeriod=30
Sep 05 02:54:05.041371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:05.041332    2560 generic.go:356] "Generic (PLEG): container finished" podID="eeecadba-c70b-4516-ba4f-393220473c7b" containerID="863d86ccef6544edf5b7301521f1fd8f790f50dbdc2f8080959448fca7966759" exitCode=2
Sep 05 02:54:05.041371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:05.041376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerDied","Data":"863d86ccef6544edf5b7301521f1fd8f790f50dbdc2f8080959448fca7966759"}
Sep 05 02:54:05.995310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:05.995272    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 05 02:54:06.975236 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:06.975206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:54:07.054891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.054856    2560 generic.go:356] "Generic (PLEG): container finished" podID="eeecadba-c70b-4516-ba4f-393220473c7b" containerID="88e09b7b3c56ffa8dac862d921499bcc4f3369b04760e15df409296b1f89e3f7" exitCode=0
Sep 05 02:54:07.055373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.054921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerDied","Data":"88e09b7b3c56ffa8dac862d921499bcc4f3369b04760e15df409296b1f89e3f7"}
Sep 05 02:54:07.165071 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.165048    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:54:07.231374 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231315    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location\") pod \"eeecadba-c70b-4516-ba4f-393220473c7b\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") "
Sep 05 02:54:07.231374 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231366    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config\") pod \"eeecadba-c70b-4516-ba4f-393220473c7b\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") "
Sep 05 02:54:07.231550 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231397    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wckbg\" (UniqueName: \"kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg\") pod \"eeecadba-c70b-4516-ba4f-393220473c7b\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") "
Sep 05 02:54:07.231550 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231420    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") pod \"eeecadba-c70b-4516-ba4f-393220473c7b\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") "
Sep 05 02:54:07.231550 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert\") pod \"eeecadba-c70b-4516-ba4f-393220473c7b\" (UID: \"eeecadba-c70b-4516-ba4f-393220473c7b\") "
Sep 05 02:54:07.231824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231798    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eeecadba-c70b-4516-ba4f-393220473c7b" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:07.231824 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eeecadba-c70b-4516-ba4f-393220473c7b" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:07.231914 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.231848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eeecadba-c70b-4516-ba4f-393220473c7b" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:07.233510 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.233486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg" (OuterVolumeSpecName: "kube-api-access-wckbg") pod "eeecadba-c70b-4516-ba4f-393220473c7b" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b"). InnerVolumeSpecName "kube-api-access-wckbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:07.233604 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.233512    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eeecadba-c70b-4516-ba4f-393220473c7b" (UID: "eeecadba-c70b-4516-ba4f-393220473c7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:07.332108 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.332081    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:07.332108 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.332106    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wckbg\" (UniqueName: \"kubernetes.io/projected/eeecadba-c70b-4516-ba4f-393220473c7b-kube-api-access-wckbg\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:07.332250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.332121    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeecadba-c70b-4516-ba4f-393220473c7b-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:07.332250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.332135    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeecadba-c70b-4516-ba4f-393220473c7b-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:07.332250 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:07.332147    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeecadba-c70b-4516-ba4f-393220473c7b-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:08.061502 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.061474    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"
Sep 05 02:54:08.061908 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.061477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws" event={"ID":"eeecadba-c70b-4516-ba4f-393220473c7b","Type":"ContainerDied","Data":"406ab87eeb1e44602b5f0f098446af38de7b673d825f16efc01fd5cbe73f79e9"}
Sep 05 02:54:08.061908 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.061610    2560 scope.go:117] "RemoveContainer" containerID="863d86ccef6544edf5b7301521f1fd8f790f50dbdc2f8080959448fca7966759"
Sep 05 02:54:08.071818 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.071737    2560 scope.go:117] "RemoveContainer" containerID="88e09b7b3c56ffa8dac862d921499bcc4f3369b04760e15df409296b1f89e3f7"
Sep 05 02:54:08.078933 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.078914    2560 scope.go:117] "RemoveContainer" containerID="66a77003f698d07e67d257068b5a00f3aa557d9478488f0135199ada9579763d"
Sep 05 02:54:08.094222 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.094204    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:54:08.101581 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.101561    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7ftws"]
Sep 05 02:54:08.439341 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.439305    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:54:08.439707 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.439547    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container" containerID="cri-o://a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7" gracePeriod=30
Sep 05 02:54:08.439707 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.439598    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kube-rbac-proxy" containerID="cri-o://5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002" gracePeriod=30
Sep 05 02:54:08.969806 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:08.969769    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" path="/var/lib/kubelet/pods/eeecadba-c70b-4516-ba4f-393220473c7b/volumes"
Sep 05 02:54:09.068122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:09.068095    2560 generic.go:356] "Generic (PLEG): container finished" podID="768fc060-e005-4e74-853f-ca17691a93fe" containerID="5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002" exitCode=2
Sep 05 02:54:09.068486 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:09.068175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerDied","Data":"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"}
Sep 05 02:54:10.551992 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:10.551952    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 05 02:54:12.982955 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:12.982931    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:54:13.073315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert\") pod \"768fc060-e005-4e74-853f-ca17691a93fe\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") "
Sep 05 02:54:13.073432 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config\") pod \"768fc060-e005-4e74-853f-ca17691a93fe\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") "
Sep 05 02:54:13.073432 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073410    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wggxv\" (UniqueName: \"kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv\") pod \"768fc060-e005-4e74-853f-ca17691a93fe\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") "
Sep 05 02:54:13.073432 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls\") pod \"768fc060-e005-4e74-853f-ca17691a93fe\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") "
Sep 05 02:54:13.073570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073454    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location\") pod \"768fc060-e005-4e74-853f-ca17691a93fe\" (UID: \"768fc060-e005-4e74-853f-ca17691a93fe\") "
Sep 05 02:54:13.073636 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073616    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "768fc060-e005-4e74-853f-ca17691a93fe" (UID: "768fc060-e005-4e74-853f-ca17691a93fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:13.073844 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "768fc060-e005-4e74-853f-ca17691a93fe" (UID: "768fc060-e005-4e74-853f-ca17691a93fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:13.073844 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.073839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "768fc060-e005-4e74-853f-ca17691a93fe" (UID: "768fc060-e005-4e74-853f-ca17691a93fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:13.075448 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.075425    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "768fc060-e005-4e74-853f-ca17691a93fe" (UID: "768fc060-e005-4e74-853f-ca17691a93fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:13.075536 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.075482    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv" (OuterVolumeSpecName: "kube-api-access-wggxv") pod "768fc060-e005-4e74-853f-ca17691a93fe" (UID: "768fc060-e005-4e74-853f-ca17691a93fe"). InnerVolumeSpecName "kube-api-access-wggxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:13.088375 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.088351    2560 generic.go:356] "Generic (PLEG): container finished" podID="768fc060-e005-4e74-853f-ca17691a93fe" containerID="a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7" exitCode=0
Sep 05 02:54:13.088500 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.088420    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"
Sep 05 02:54:13.088590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.088422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerDied","Data":"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"}
Sep 05 02:54:13.088590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.088549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh" event={"ID":"768fc060-e005-4e74-853f-ca17691a93fe","Type":"ContainerDied","Data":"5307a6c3f765cbb9472dc478ef6be0413a0bda11de70937a349b471edd97f57d"}
Sep 05 02:54:13.088590 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.088578    2560 scope.go:117] "RemoveContainer" containerID="5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"
Sep 05 02:54:13.097300 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.097283    2560 scope.go:117] "RemoveContainer" containerID="a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"
Sep 05 02:54:13.104482 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.104466    2560 scope.go:117] "RemoveContainer" containerID="7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"
Sep 05 02:54:13.112816 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.112797    2560 scope.go:117] "RemoveContainer" containerID="5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"
Sep 05 02:54:13.113082 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:54:13.113066    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002\": container with ID starting with 5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002 not found: ID does not exist" containerID="5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"
Sep 05 02:54:13.113128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113089    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002"} err="failed to get container status \"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002\": rpc error: code = NotFound desc = could not find container \"5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002\": container with ID starting with 5d04078dcf66cc669380c44d9226844b0f9508443cde31941e76b6f3e569f002 not found: ID does not exist"
Sep 05 02:54:13.113128 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113104    2560 scope.go:117] "RemoveContainer" containerID="a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"
Sep 05 02:54:13.113359 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113341    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:54:13.113404 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:54:13.113345    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7\": container with ID starting with a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7 not found: ID does not exist" containerID="a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"
Sep 05 02:54:13.113404 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113390    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7"} err="failed to get container status \"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7\": rpc error: code = NotFound desc = could not find container \"a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7\": container with ID starting with a45434ca735dc85d1cfba448e0d5bac4e2721be73232637cd880b4217b552da7 not found: ID does not exist"
Sep 05 02:54:13.113477 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113405    2560 scope.go:117] "RemoveContainer" containerID="7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"
Sep 05 02:54:13.113636 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:54:13.113618    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553\": container with ID starting with 7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553 not found: ID does not exist" containerID="7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"
Sep 05 02:54:13.113691 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.113645    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553"} err="failed to get container status \"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553\": rpc error: code = NotFound desc = could not find container \"7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553\": container with ID starting with 7775925bf3accdc33b1c4af13eba09a52670646d6c34f543cddbe55f7490e553 not found: ID does not exist"
Sep 05 02:54:13.118209 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.118184    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-cmmdh"]
Sep 05 02:54:13.174103 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.174081    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.174185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.174109    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggxv\" (UniqueName: \"kubernetes.io/projected/768fc060-e005-4e74-853f-ca17691a93fe-kube-api-access-wggxv\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.174185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.174125    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768fc060-e005-4e74-853f-ca17691a93fe-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.174185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.174143    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768fc060-e005-4e74-853f-ca17691a93fe-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.174185 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.174156    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768fc060-e005-4e74-853f-ca17691a93fe-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.517505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.517468    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:54:13.517941 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.517926    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container"
Sep 05 02:54:13.518010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.517954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container"
Sep 05 02:54:13.518010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.517975    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="storage-initializer"
Sep 05 02:54:13.518010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.517984    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="storage-initializer"
Sep 05 02:54:13.518010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518008    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518017    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518030    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="storage-initializer"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518038    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="storage-initializer"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518048    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518056    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518067    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518075    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518085    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518104    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="storage-initializer"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518113    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="storage-initializer"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518124    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container"
Sep 05 02:54:13.518145 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518132    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518197    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518210    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kserve-container"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518220    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kube-rbac-proxy"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518227    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeecadba-c70b-4516-ba4f-393220473c7b" containerName="kserve-container"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518236    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="768fc060-e005-4e74-853f-ca17691a93fe" containerName="kserve-container"
Sep 05 02:54:13.518539 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.518244    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="794278dc-91ba-4e39-8b3c-b157aff68337" containerName="kube-rbac-proxy"
Sep 05 02:54:13.520586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.520566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.523143 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.523119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:54:13.523260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.523119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 05 02:54:13.523260 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.523239    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:54:13.524296 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.524270    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:54:13.524296 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.524284    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:54:13.524458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.524400    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:54:13.524514 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.524499    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 05 02:54:13.531687 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.531660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:54:13.577956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.577934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.578062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.577973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.578062 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.577999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.578140 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.578084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfsm\" (UniqueName: \"kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.578187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.578175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.603073 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.603050    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:54:13.605267 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.605251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.608052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.608033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 05 02:54:13.617266 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.617246    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:54:13.678778 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.678879 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.678876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.679158 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:54:13.678980    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 05 02:54:13.679158 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:54:13.679073    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls podName:ebc13762-0b35-4a8c-8dfe-51d43f5cee5a nodeName:}" failed. No retries permitted until 2026-09-05 02:54:14.179052634 +0000 UTC m=+2197.818264428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls") pod "isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 05 02:54:13.679158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.679158 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd62z\" (UniqueName: \"kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.679353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.679353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.679353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfsm\" (UniqueName: \"kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.679494 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.679781 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.679735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.689680 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.689658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfsm\" (UniqueName: \"kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:13.780169 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780169 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd62z\" (UniqueName: \"kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780614 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780786 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.780861 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.780797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.782408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.782390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.788944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.788924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd62z\" (UniqueName: \"kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z\") pod \"isvc-canary-stop-predictor-79f67b4d78-2fblz\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:13.914464 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:13.914439    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:14.184706 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.184675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:14.187238 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.187212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:14.188608 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.188585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:54:14.190796 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:54:14.190773    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3cbd0d_6965_467f_b764_3f217849fa2f.slice/crio-5c5c12e1c6b84b7792e87ad43c49c2f43c2fdeb62e9c245c9645b1fe730f9e86 WatchSource:0}: Error finding container 5c5c12e1c6b84b7792e87ad43c49c2f43c2fdeb62e9c245c9645b1fe730f9e86: Status 404 returned error can't find the container with id 5c5c12e1c6b84b7792e87ad43c49c2f43c2fdeb62e9c245c9645b1fe730f9e86
Sep 05 02:54:14.432938 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.432901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:14.716025 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.715937    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:54:14.719640 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:54:14.719609    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc13762_0b35_4a8c_8dfe_51d43f5cee5a.slice/crio-bafc3f18ceeab84057bbea635cbf8b3741a38e22f2c254865ba9130c1610db10 WatchSource:0}: Error finding container bafc3f18ceeab84057bbea635cbf8b3741a38e22f2c254865ba9130c1610db10: Status 404 returned error can't find the container with id bafc3f18ceeab84057bbea635cbf8b3741a38e22f2c254865ba9130c1610db10
Sep 05 02:54:14.970959 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:14.970878    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768fc060-e005-4e74-853f-ca17691a93fe" path="/var/lib/kubelet/pods/768fc060-e005-4e74-853f-ca17691a93fe/volumes"
Sep 05 02:54:15.098766 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:15.098711    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerStarted","Data":"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"}
Sep 05 02:54:15.098945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:15.098798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerStarted","Data":"bafc3f18ceeab84057bbea635cbf8b3741a38e22f2c254865ba9130c1610db10"}
Sep 05 02:54:15.100384 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:15.100361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerStarted","Data":"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"}
Sep 05 02:54:15.100518 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:15.100412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerStarted","Data":"5c5c12e1c6b84b7792e87ad43c49c2f43c2fdeb62e9c245c9645b1fe730f9e86"}
Sep 05 02:54:19.119271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:19.119196    2560 generic.go:356] "Generic (PLEG): container finished" podID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerID="9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791" exitCode=0
Sep 05 02:54:19.119651 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:19.119267    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerDied","Data":"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"}
Sep 05 02:54:19.120938 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:19.120916    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerID="f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed" exitCode=0
Sep 05 02:54:19.121053 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:19.120990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerDied","Data":"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"}
Sep 05 02:54:20.128909 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.128854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerStarted","Data":"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"}
Sep 05 02:54:20.129342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.128923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerStarted","Data":"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"}
Sep 05 02:54:20.129342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.129213    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:20.129342 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.129271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:20.130912 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.130877    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:20.131262 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.131241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerStarted","Data":"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"}
Sep 05 02:54:20.131375 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.131286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerStarted","Data":"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"}
Sep 05 02:54:20.131547 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.131527    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:20.131617 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.131574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:20.132647 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.132622    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:54:20.148656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.148614    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podStartSLOduration=3.168216733 podStartE2EDuration="7.148600778s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980384045s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:20.146436413 +0000 UTC m=+2203.785648232" watchObservedRunningTime="2026-09-05 02:54:20.148600778 +0000 UTC m=+2203.787812578"
Sep 05 02:54:20.163711 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:20.163666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podStartSLOduration=3.195668966 podStartE2EDuration="7.163651242s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.967982276s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:20.16330336 +0000 UTC m=+2203.802515187" watchObservedRunningTime="2026-09-05 02:54:20.163651242 +0000 UTC m=+2203.802863054"
Sep 05 02:54:21.136946 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:21.136906    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:54:21.137410 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:21.136906    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:26.141130 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:26.141099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:54:26.141830 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:26.141795    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:54:26.142134 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:26.142111    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:54:26.142587 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:26.142561    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:36.142271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:36.142233    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:54:36.142625 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:36.142545    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:46.141851 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:46.141807    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:54:46.143028 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:46.143002    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:56.142567 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:56.142520    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:54:56.143052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:54:56.142520    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:55:06.141815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:06.141700    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:55:06.143216 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:06.143183    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:55:16.141686 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:16.141646    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:55:16.142877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:16.142849    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:55:26.142535 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:26.142490    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:55:26.142945 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:26.142490    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:55:36.142503 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:36.142474    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:55:36.143269 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:36.143249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:55:43.552192 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.552157    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:55:43.552688 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.552378    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" containerID="cri-o://cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d" gracePeriod=30
Sep 05 02:55:43.552688 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.552430    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kube-rbac-proxy" containerID="cri-o://ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111" gracePeriod=30
Sep 05 02:55:43.591826 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.591798    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:55:43.592031 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.592008    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" containerID="cri-o://64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978" gracePeriod=30
Sep 05 02:55:43.592109 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:43.592034    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kube-rbac-proxy" containerID="cri-o://5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f" gracePeriod=30
Sep 05 02:55:44.528124 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:44.528088    2560 generic.go:356] "Generic (PLEG): container finished" podID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerID="ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111" exitCode=2
Sep 05 02:55:44.528296 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:44.528163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerDied","Data":"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"}
Sep 05 02:55:44.530253 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:44.530230    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerID="5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f" exitCode=2
Sep 05 02:55:44.530350 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:44.530295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerDied","Data":"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"}
Sep 05 02:55:46.137251 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:46.137208    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 05 02:55:46.137635 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:46.137215    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 05 02:55:46.142585 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:46.142561    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 05 02:55:46.142672 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:46.142564    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 05 02:55:48.237524 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.237499    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:55:48.266272 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd62z\" (UniqueName: \"kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z\") pod \"8b3cbd0d-6965-467f-b764-3f217849fa2f\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") "
Sep 05 02:55:48.266414 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config\") pod \"8b3cbd0d-6965-467f-b764-3f217849fa2f\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") "
Sep 05 02:55:48.266414 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert\") pod \"8b3cbd0d-6965-467f-b764-3f217849fa2f\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") "
Sep 05 02:55:48.266414 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266366    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location\") pod \"8b3cbd0d-6965-467f-b764-3f217849fa2f\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") "
Sep 05 02:55:48.266569 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls\") pod \"8b3cbd0d-6965-467f-b764-3f217849fa2f\" (UID: \"8b3cbd0d-6965-467f-b764-3f217849fa2f\") "
Sep 05 02:55:48.266734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b3cbd0d-6965-467f-b764-3f217849fa2f" (UID: "8b3cbd0d-6965-467f-b764-3f217849fa2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:48.266734 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b3cbd0d-6965-467f-b764-3f217849fa2f" (UID: "8b3cbd0d-6965-467f-b764-3f217849fa2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:48.266911 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.266774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b3cbd0d-6965-467f-b764-3f217849fa2f" (UID: "8b3cbd0d-6965-467f-b764-3f217849fa2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:48.269882 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.269855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b3cbd0d-6965-467f-b764-3f217849fa2f" (UID: "8b3cbd0d-6965-467f-b764-3f217849fa2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:48.269999 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.269979    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z" (OuterVolumeSpecName: "kube-api-access-fd62z") pod "8b3cbd0d-6965-467f-b764-3f217849fa2f" (UID: "8b3cbd0d-6965-467f-b764-3f217849fa2f"). InnerVolumeSpecName "kube-api-access-fd62z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:48.284700 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.284677    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:55:48.367164 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") pod \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") "
Sep 05 02:55:48.367164 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location\") pod \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") "
Sep 05 02:55:48.367344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367218    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config\") pod \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") "
Sep 05 02:55:48.367344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert\") pod \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") "
Sep 05 02:55:48.367344 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367305    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjfsm\" (UniqueName: \"kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm\") pod \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\" (UID: \"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a\") "
Sep 05 02:55:48.367555 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367539    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.367615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367543    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:48.367615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:48.367615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367562    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b3cbd0d-6965-467f-b764-3f217849fa2f-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.367615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367586    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:48.367615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367598    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b3cbd0d-6965-467f-b764-3f217849fa2f-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.367827 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367616    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd62z\" (UniqueName: \"kubernetes.io/projected/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-api-access-fd62z\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.367827 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.367629    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b3cbd0d-6965-467f-b764-3f217849fa2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.369271 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.369248    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:48.369434 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.369414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm" (OuterVolumeSpecName: "kube-api-access-bjfsm") pod "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" (UID: "ebc13762-0b35-4a8c-8dfe-51d43f5cee5a"). InnerVolumeSpecName "kube-api-access-bjfsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:48.468570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.468547    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.468570 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.468568    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.468710 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.468577    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfsm\" (UniqueName: \"kubernetes.io/projected/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kube-api-access-bjfsm\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.468710 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.468586    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.468710 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.468595    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:55:48.551675 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.551644    2560 generic.go:356] "Generic (PLEG): container finished" podID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerID="cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d" exitCode=0
Sep 05 02:55:48.551833 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.551713    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"
Sep 05 02:55:48.551833 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.551728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerDied","Data":"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"}
Sep 05 02:55:48.551833 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.551816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp" event={"ID":"ebc13762-0b35-4a8c-8dfe-51d43f5cee5a","Type":"ContainerDied","Data":"bafc3f18ceeab84057bbea635cbf8b3741a38e22f2c254865ba9130c1610db10"}
Sep 05 02:55:48.551985 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.551859    2560 scope.go:117] "RemoveContainer" containerID="ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"
Sep 05 02:55:48.554054 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.554030    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerID="64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978" exitCode=0
Sep 05 02:55:48.554173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.554101    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"
Sep 05 02:55:48.554173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.554101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerDied","Data":"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"}
Sep 05 02:55:48.554173 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.554168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz" event={"ID":"8b3cbd0d-6965-467f-b764-3f217849fa2f","Type":"ContainerDied","Data":"5c5c12e1c6b84b7792e87ad43c49c2f43c2fdeb62e9c245c9645b1fe730f9e86"}
Sep 05 02:55:48.560714 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.560697    2560 scope.go:117] "RemoveContainer" containerID="cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"
Sep 05 02:55:48.570698 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.570681    2560 scope.go:117] "RemoveContainer" containerID="9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"
Sep 05 02:55:48.580808 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.579456    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:55:48.582074 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.582045    2560 scope.go:117] "RemoveContainer" containerID="ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"
Sep 05 02:55:48.582791 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.582771    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111\": container with ID starting with ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111 not found: ID does not exist" containerID="ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"
Sep 05 02:55:48.582894 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.582797    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111"} err="failed to get container status \"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111\": rpc error: code = NotFound desc = could not find container \"ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111\": container with ID starting with ee0f02166bff3b81199f3fa743ec3b8bdab2b97c1848c17588ad7bfe52d7a111 not found: ID does not exist"
Sep 05 02:55:48.582894 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.582814    2560 scope.go:117] "RemoveContainer" containerID="cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"
Sep 05 02:55:48.583108 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.583091    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d\": container with ID starting with cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d not found: ID does not exist" containerID="cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"
Sep 05 02:55:48.583178 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.583113    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d"} err="failed to get container status \"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d\": rpc error: code = NotFound desc = could not find container \"cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d\": container with ID starting with cecc0c4097b8329625c48f7a579ac79eda8e867581634ce336ad4e36ceb6b51d not found: ID does not exist"
Sep 05 02:55:48.583178 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.583128    2560 scope.go:117] "RemoveContainer" containerID="9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"
Sep 05 02:55:48.583277 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.583260    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-ng5qp"]
Sep 05 02:55:48.583438 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.583420    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791\": container with ID starting with 9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791 not found: ID does not exist" containerID="9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"
Sep 05 02:55:48.583493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.583445    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791"} err="failed to get container status \"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791\": rpc error: code = NotFound desc = could not find container \"9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791\": container with ID starting with 9e01d31ebdb1870b903a5f5abbacc0d262add924d855669a4e5c2f6f2a490791 not found: ID does not exist"
Sep 05 02:55:48.583493 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.583458    2560 scope.go:117] "RemoveContainer" containerID="5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"
Sep 05 02:55:48.591144 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.591115    2560 scope.go:117] "RemoveContainer" containerID="64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"
Sep 05 02:55:48.598498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.598483    2560 scope.go:117] "RemoveContainer" containerID="f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"
Sep 05 02:55:48.598846 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.598826    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:55:48.603800 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.603779    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2fblz"]
Sep 05 02:55:48.607117 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.607096    2560 scope.go:117] "RemoveContainer" containerID="5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"
Sep 05 02:55:48.607417 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.607399    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f\": container with ID starting with 5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f not found: ID does not exist" containerID="5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"
Sep 05 02:55:48.607474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.607422    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f"} err="failed to get container status \"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f\": rpc error: code = NotFound desc = could not find container \"5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f\": container with ID starting with 5cad8c03935d02de10e393dd20e0b9c34192a57e6b0105d44cee69c3b837cb3f not found: ID does not exist"
Sep 05 02:55:48.607474 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.607438    2560 scope.go:117] "RemoveContainer" containerID="64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"
Sep 05 02:55:48.607737 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.607717    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978\": container with ID starting with 64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978 not found: ID does not exist" containerID="64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"
Sep 05 02:55:48.607737 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.607757    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978"} err="failed to get container status \"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978\": rpc error: code = NotFound desc = could not find container \"64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978\": container with ID starting with 64298f20a8a2efd76100af169c5fb3df36fb4dafb8d8582762376dc7b7f4f978 not found: ID does not exist"
Sep 05 02:55:48.607737 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.607779    2560 scope.go:117] "RemoveContainer" containerID="f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"
Sep 05 02:55:48.608053 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:55:48.608029    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed\": container with ID starting with f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed not found: ID does not exist" containerID="f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"
Sep 05 02:55:48.608107 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.608047    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed"} err="failed to get container status \"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed\": rpc error: code = NotFound desc = could not find container \"f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed\": container with ID starting with f077a308d68e74ff26b9265b7dbfef88caf55768aa2d3e26195547fbcbfa22ed not found: ID does not exist"
Sep 05 02:55:48.970650 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.970618    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" path="/var/lib/kubelet/pods/8b3cbd0d-6965-467f-b764-3f217849fa2f/volumes"
Sep 05 02:55:48.971162 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:48.971146    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" path="/var/lib/kubelet/pods/ebc13762-0b35-4a8c-8dfe-51d43f5cee5a/volumes"
Sep 05 02:55:54.676599 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676566    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676881    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676892    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676900    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="storage-initializer"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676906    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="storage-initializer"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676917    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kube-rbac-proxy"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676923    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kube-rbac-proxy"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676937    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676942    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676949    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kube-rbac-proxy"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kube-rbac-proxy"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676963    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="storage-initializer"
Sep 05 02:55:54.677003 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.676968    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="storage-initializer"
Sep 05 02:55:54.677373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.677020    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kserve-container"
Sep 05 02:55:54.677373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.677029    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3cbd0d-6965-467f-b764-3f217849fa2f" containerName="kube-rbac-proxy"
Sep 05 02:55:54.677373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.677037    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kserve-container"
Sep 05 02:55:54.677373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.677043    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc13762-0b35-4a8c-8dfe-51d43f5cee5a" containerName="kube-rbac-proxy"
Sep 05 02:55:54.679989 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.679973    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.682670 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.682634    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:55:54.683790 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.683770    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 05 02:55:54.683896 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.683832    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:55:54.683963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.683902    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:55:54.683963 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.683941    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:55:54.684081 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.683972    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 05 02:55:54.684132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.684081    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:55:54.696938 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.696917    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:55:54.711802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.711781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.711894 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.711809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rk8c\" (UniqueName: \"kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.711894 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.711860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.711970 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.711910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.712010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.711985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812309 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rk8c\" (UniqueName: \"kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812417 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812787 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.812971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.812951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.813061 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.813044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.814798 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.814780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.823014 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.822996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rk8c\" (UniqueName: \"kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:54.990450 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:54.990385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:55:55.263231 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:55.263154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:55:55.266442 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:55:55.266415    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf211cb98_8b77_4156_9e30_07056fbdbbef.slice/crio-ee50510d637b6fb74d13eb9bf93a6d675bc83da38994aac01f5e72760568c98a WatchSource:0}: Error finding container ee50510d637b6fb74d13eb9bf93a6d675bc83da38994aac01f5e72760568c98a: Status 404 returned error can't find the container with id ee50510d637b6fb74d13eb9bf93a6d675bc83da38994aac01f5e72760568c98a
Sep 05 02:55:55.588205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:55.588109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerStarted","Data":"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"}
Sep 05 02:55:55.588205 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:55.588180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerStarted","Data":"ee50510d637b6fb74d13eb9bf93a6d675bc83da38994aac01f5e72760568c98a"}
Sep 05 02:55:59.607656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:59.607575    2560 generic.go:356] "Generic (PLEG): container finished" podID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerID="b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08" exitCode=0
Sep 05 02:55:59.608022 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:55:59.607649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerDied","Data":"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"}
Sep 05 02:56:00.614615 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:00.614576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerStarted","Data":"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"}
Sep 05 02:56:00.614997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:00.614641    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerStarted","Data":"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"}
Sep 05 02:56:00.614997 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:00.614889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:00.635935 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:00.635892    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" podStartSLOduration=2.7029599170000003 podStartE2EDuration="6.63587835s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.932918433s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:00.63456827 +0000 UTC m=+2304.273780103" watchObservedRunningTime="2026-09-05 02:56:00.63587835 +0000 UTC m=+2304.275090150"
Sep 05 02:56:01.618806 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:01.618774    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:07.629080 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:07.629049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:37.633429 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:37.633349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:44.657605 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:44.657573    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:56:44.658129 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:44.657816    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kserve-container" containerID="cri-o://cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326" gracePeriod=30
Sep 05 02:56:44.658129 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:44.657855    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kube-rbac-proxy" containerID="cri-o://8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04" gracePeriod=30
Sep 05 02:56:44.815117 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:44.814962    2560 generic.go:356] "Generic (PLEG): container finished" podID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerID="8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04" exitCode=2
Sep 05 02:56:44.815117 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:44.815102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerDied","Data":"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"}
Sep 05 02:56:47.624422 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:47.624382    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 05 02:56:47.629820 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:47.629795    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 05 02:56:49.855580 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.855549    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:56:49.858115 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.858093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:49.860354 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.860334    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 05 02:56:49.860458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.860417    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 05 02:56:49.867838 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.867817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:56:49.924956 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.924922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vc4\" (UniqueName: \"kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:49.925068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.924959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:49.925068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.924977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:49.925068 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.925015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:49.925195 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:49.925086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.025588 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vc4\" (UniqueName: \"kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.025765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.025765 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.025906 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.025968 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.026081 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.025963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.026526 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.026501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.026617 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.026501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.028150 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.028130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.035825 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.035800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vc4\" (UniqueName: \"kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7gq59\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.169380 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.169356    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:50.200877 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.200850    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:50.227756 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.227709    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rk8c\" (UniqueName: \"kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c\") pod \"f211cb98-8b77-4156-9e30-07056fbdbbef\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") "
Sep 05 02:56:50.227988 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.227970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location\") pod \"f211cb98-8b77-4156-9e30-07056fbdbbef\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") "
Sep 05 02:56:50.228537 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.228495    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f211cb98-8b77-4156-9e30-07056fbdbbef" (UID: "f211cb98-8b77-4156-9e30-07056fbdbbef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:50.228803 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.228697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls\") pod \"f211cb98-8b77-4156-9e30-07056fbdbbef\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") "
Sep 05 02:56:50.229332 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.229000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config\") pod \"f211cb98-8b77-4156-9e30-07056fbdbbef\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") "
Sep 05 02:56:50.229332 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.229062    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert\") pod \"f211cb98-8b77-4156-9e30-07056fbdbbef\" (UID: \"f211cb98-8b77-4156-9e30-07056fbdbbef\") "
Sep 05 02:56:50.229496 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.229365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f211cb98-8b77-4156-9e30-07056fbdbbef" (UID: "f211cb98-8b77-4156-9e30-07056fbdbbef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:56:50.229557 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.229539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f211cb98-8b77-4156-9e30-07056fbdbbef" (UID: "f211cb98-8b77-4156-9e30-07056fbdbbef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:56:50.229950 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.229930    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:56:50.230096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.230060    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f211cb98-8b77-4156-9e30-07056fbdbbef-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:56:50.230096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.230079    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:56:50.233035 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.233008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f211cb98-8b77-4156-9e30-07056fbdbbef" (UID: "f211cb98-8b77-4156-9e30-07056fbdbbef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:56:50.233132 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.233094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c" (OuterVolumeSpecName: "kube-api-access-9rk8c") pod "f211cb98-8b77-4156-9e30-07056fbdbbef" (UID: "f211cb98-8b77-4156-9e30-07056fbdbbef"). InnerVolumeSpecName "kube-api-access-9rk8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:56:50.330707 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.330683    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rk8c\" (UniqueName: \"kubernetes.io/projected/f211cb98-8b77-4156-9e30-07056fbdbbef-kube-api-access-9rk8c\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:56:50.330707 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.330705    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f211cb98-8b77-4156-9e30-07056fbdbbef-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:56:50.445431 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.445407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:56:50.447346 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:56:50.447322    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8c862f_3ca8_4acf_b532_65b676f782ca.slice/crio-7034824d6effbf61b74cb7a456424cce84b5ebb33baf5880403af08a9c53aea0 WatchSource:0}: Error finding container 7034824d6effbf61b74cb7a456424cce84b5ebb33baf5880403af08a9c53aea0: Status 404 returned error can't find the container with id 7034824d6effbf61b74cb7a456424cce84b5ebb33baf5880403af08a9c53aea0
Sep 05 02:56:50.842467 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.842385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerStarted","Data":"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"}
Sep 05 02:56:50.842625 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.842462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerStarted","Data":"7034824d6effbf61b74cb7a456424cce84b5ebb33baf5880403af08a9c53aea0"}
Sep 05 02:56:50.844426 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.844397    2560 generic.go:356] "Generic (PLEG): container finished" podID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerID="cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326" exitCode=0
Sep 05 02:56:50.844568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.844440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"
Sep 05 02:56:50.844568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.844485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerDied","Data":"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"}
Sep 05 02:56:50.844568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.844538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg" event={"ID":"f211cb98-8b77-4156-9e30-07056fbdbbef","Type":"ContainerDied","Data":"ee50510d637b6fb74d13eb9bf93a6d675bc83da38994aac01f5e72760568c98a"}
Sep 05 02:56:50.844568 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.844566    2560 scope.go:117] "RemoveContainer" containerID="8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"
Sep 05 02:56:50.859024 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.859005    2560 scope.go:117] "RemoveContainer" containerID="cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"
Sep 05 02:56:50.866889 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.866868    2560 scope.go:117] "RemoveContainer" containerID="b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"
Sep 05 02:56:50.874448 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.874426    2560 scope.go:117] "RemoveContainer" containerID="8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"
Sep 05 02:56:50.874728 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:56:50.874705    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04\": container with ID starting with 8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04 not found: ID does not exist" containerID="8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"
Sep 05 02:56:50.874846 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.874735    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04"} err="failed to get container status \"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04\": rpc error: code = NotFound desc = could not find container \"8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04\": container with ID starting with 8675715d54aed128c8ed3077b1ca79a497d009b64d5c15b0a99f0b6e96371a04 not found: ID does not exist"
Sep 05 02:56:50.874846 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.874774    2560 scope.go:117] "RemoveContainer" containerID="cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"
Sep 05 02:56:50.875047 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:56:50.875027    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326\": container with ID starting with cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326 not found: ID does not exist" containerID="cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"
Sep 05 02:56:50.875112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.875051    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326"} err="failed to get container status \"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326\": rpc error: code = NotFound desc = could not find container \"cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326\": container with ID starting with cd946bc8da60e35fb015171a317c61c9a035b7ec4d95b9f980152cf8a8e38326 not found: ID does not exist"
Sep 05 02:56:50.875112 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.875064    2560 scope.go:117] "RemoveContainer" containerID="b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"
Sep 05 02:56:50.875284 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:56:50.875264    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08\": container with ID starting with b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08 not found: ID does not exist" containerID="b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"
Sep 05 02:56:50.875347 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.875289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08"} err="failed to get container status \"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08\": rpc error: code = NotFound desc = could not find container \"b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08\": container with ID starting with b7060bca6f64792ac189350b20895e5988b1b3f1ff79dae747a2ba6634dffa08 not found: ID does not exist"
Sep 05 02:56:50.881001 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.880977    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:56:50.884524 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.884501    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nllrg"]
Sep 05 02:56:50.970529 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:50.970500    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" path="/var/lib/kubelet/pods/f211cb98-8b77-4156-9e30-07056fbdbbef/volumes"
Sep 05 02:56:54.865281 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:54.865250    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerID="870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22" exitCode=0
Sep 05 02:56:54.865624 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:54.865303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerDied","Data":"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"}
Sep 05 02:56:55.872810 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:55.872777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerStarted","Data":"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"}
Sep 05 02:56:55.873183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:55.872834    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerStarted","Data":"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"}
Sep 05 02:56:55.873183 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:55.873048    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:55.873272 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:55.873237    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:56:55.894578 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:56:55.894533    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" podStartSLOduration=3.196178828 podStartE2EDuration="6.894520289s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698341461s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:55.892824211 +0000 UTC m=+2359.532036037" watchObservedRunningTime="2026-09-05 02:56:55.894520289 +0000 UTC m=+2359.533732089"
Sep 05 02:57:01.888566 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:01.888531    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:57:31.893160 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:31.893126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:57:36.969077 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:36.969050    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:57:36.977313 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:36.977289    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 02:57:39.909460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:39.909428    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:57:39.909888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:39.909654    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kserve-container" containerID="cri-o://4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289" gracePeriod=30
Sep 05 02:57:39.909888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:39.909727    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kube-rbac-proxy" containerID="cri-o://6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd" gracePeriod=30
Sep 05 02:57:40.067669 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:40.067639    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerID="6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd" exitCode=2
Sep 05 02:57:40.067831 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:40.067713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerDied","Data":"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"}
Sep 05 02:57:41.883475 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:41.883424    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 05 02:57:45.097151 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097116    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097438    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kube-rbac-proxy"
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097449    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kube-rbac-proxy"
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097458    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kserve-container"
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097464    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kserve-container"
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097471    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="storage-initializer"
Sep 05 02:57:45.097498 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097478    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="storage-initializer"
Sep 05 02:57:45.097694 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097529    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kube-rbac-proxy"
Sep 05 02:57:45.097694 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.097539    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f211cb98-8b77-4156-9e30-07056fbdbbef" containerName="kserve-container"
Sep 05 02:57:45.100692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.100676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.103361 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.103340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 05 02:57:45.103815 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.103799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:57:45.112891 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.112870    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:57:45.254371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.254297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.254371 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.254356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.254595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.254415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mb4\" (UniqueName: \"kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.254595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.254443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.254595 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.254523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.345285 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.345257    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:57:45.355317 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.355317 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.355458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mb4\" (UniqueName: \"kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.355458 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:57:45.355438    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 05 02:57:45.355458 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.355628 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:57:45.355502    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls podName:b48af8f0-3687-480b-b7e3-9635d6763944 nodeName:}" failed. No retries permitted until 2026-09-05 02:57:45.855481383 +0000 UTC m=+2409.494693175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-fkvff" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 05 02:57:45.355628 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.355858 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.355830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.356126 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.356109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.356207 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.356184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.364971 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.364945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mb4\" (UniqueName: \"kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.456097 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456070    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config\") pod \"6a8c862f-3ca8-4acf-b532-65b676f782ca\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") "
Sep 05 02:57:45.456265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86vc4\" (UniqueName: \"kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4\") pod \"6a8c862f-3ca8-4acf-b532-65b676f782ca\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") "
Sep 05 02:57:45.456265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls\") pod \"6a8c862f-3ca8-4acf-b532-65b676f782ca\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") "
Sep 05 02:57:45.456265 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456211    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location\") pod \"6a8c862f-3ca8-4acf-b532-65b676f782ca\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") "
Sep 05 02:57:45.456447 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert\") pod \"6a8c862f-3ca8-4acf-b532-65b676f782ca\" (UID: \"6a8c862f-3ca8-4acf-b532-65b676f782ca\") "
Sep 05 02:57:45.456505 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456432    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a8c862f-3ca8-4acf-b532-65b676f782ca" (UID: "6a8c862f-3ca8-4acf-b532-65b676f782ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:45.456607 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456581    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:57:45.456664 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456582    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a8c862f-3ca8-4acf-b532-65b676f782ca" (UID: "6a8c862f-3ca8-4acf-b532-65b676f782ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:45.456722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.456660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a8c862f-3ca8-4acf-b532-65b676f782ca" (UID: "6a8c862f-3ca8-4acf-b532-65b676f782ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:45.458290 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.458266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a8c862f-3ca8-4acf-b532-65b676f782ca" (UID: "6a8c862f-3ca8-4acf-b532-65b676f782ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:57:45.458389 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.458293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4" (OuterVolumeSpecName: "kube-api-access-86vc4") pod "6a8c862f-3ca8-4acf-b532-65b676f782ca" (UID: "6a8c862f-3ca8-4acf-b532-65b676f782ca"). InnerVolumeSpecName "kube-api-access-86vc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:57:45.557804 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.557774    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a8c862f-3ca8-4acf-b532-65b676f782ca-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:57:45.557804 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.557799    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vc4\" (UniqueName: \"kubernetes.io/projected/6a8c862f-3ca8-4acf-b532-65b676f782ca-kube-api-access-86vc4\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:57:45.557804 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.557811    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8c862f-3ca8-4acf-b532-65b676f782ca-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:57:45.558010 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.557821    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a8c862f-3ca8-4acf-b532-65b676f782ca-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:57:45.860409 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.860379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:45.862692 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:45.862665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fkvff\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:46.013230 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.013183    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:57:46.096460 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.096431    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerID="4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289" exitCode=0
Sep 05 02:57:46.096586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.096484    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"
Sep 05 02:57:46.096586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.096501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerDied","Data":"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"}
Sep 05 02:57:46.096586 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.096559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59" event={"ID":"6a8c862f-3ca8-4acf-b532-65b676f782ca","Type":"ContainerDied","Data":"7034824d6effbf61b74cb7a456424cce84b5ebb33baf5880403af08a9c53aea0"}
Sep 05 02:57:46.096705 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.096588    2560 scope.go:117] "RemoveContainer" containerID="6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"
Sep 05 02:57:46.105373 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.105350    2560 scope.go:117] "RemoveContainer" containerID="4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"
Sep 05 02:57:46.112973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.112963    2560 scope.go:117] "RemoveContainer" containerID="870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"
Sep 05 02:57:46.120394 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.120359    2560 scope.go:117] "RemoveContainer" containerID="6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"
Sep 05 02:57:46.120612 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:57:46.120592    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd\": container with ID starting with 6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd not found: ID does not exist" containerID="6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"
Sep 05 02:57:46.120677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.120619    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd"} err="failed to get container status \"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd\": rpc error: code = NotFound desc = could not find container \"6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd\": container with ID starting with 6810cf82f15ef90802200f2a71bcd9b1e445568cb3c0757ede2b1f1816bfb8fd not found: ID does not exist"
Sep 05 02:57:46.120677 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.120635    2560 scope.go:117] "RemoveContainer" containerID="4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"
Sep 05 02:57:46.120877 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:57:46.120859    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289\": container with ID starting with 4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289 not found: ID does not exist" containerID="4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"
Sep 05 02:57:46.120944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.120881    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289"} err="failed to get container status \"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289\": rpc error: code = NotFound desc = could not find container \"4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289\": container with ID starting with 4094f2dab420754b5e59d91aabbf3dc397a15a6b3e5860e14bc43c1635cd0289 not found: ID does not exist"
Sep 05 02:57:46.120944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.120900    2560 scope.go:117] "RemoveContainer" containerID="870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"
Sep 05 02:57:46.121140 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:57:46.121121    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22\": container with ID starting with 870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22 not found: ID does not exist" containerID="870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"
Sep 05 02:57:46.121202 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.121143    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22"} err="failed to get container status \"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22\": rpc error: code = NotFound desc = could not find container \"870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22\": container with ID starting with 870e66d5e02451078799480f3fea24fdb693fa0b565ab061c7aadf7c6dea0b22 not found: ID does not exist"
Sep 05 02:57:46.121308 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.121289    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:57:46.125355 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.125335    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7gq59"]
Sep 05 02:57:46.287618 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.287596    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:57:46.289663 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:57:46.289634    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48af8f0_3687_480b_b7e3_9635d6763944.slice/crio-0f5119a9fbc49d9e698f673f3ab9dbfb11ee2a1dad451ebe0fff62547570c853 WatchSource:0}: Error finding container 0f5119a9fbc49d9e698f673f3ab9dbfb11ee2a1dad451ebe0fff62547570c853: Status 404 returned error can't find the container with id 0f5119a9fbc49d9e698f673f3ab9dbfb11ee2a1dad451ebe0fff62547570c853
Sep 05 02:57:46.970363 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:46.970328    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" path="/var/lib/kubelet/pods/6a8c862f-3ca8-4acf-b532-65b676f782ca/volumes"
Sep 05 02:57:47.101906 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:47.101871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerStarted","Data":"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"}
Sep 05 02:57:47.102093 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:47.101945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerStarted","Data":"0f5119a9fbc49d9e698f673f3ab9dbfb11ee2a1dad451ebe0fff62547570c853"}
Sep 05 02:57:51.122144 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:51.122108    2560 generic.go:356] "Generic (PLEG): container finished" podID="b48af8f0-3687-480b-b7e3-9635d6763944" containerID="a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac" exitCode=0
Sep 05 02:57:51.122504 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:51.122180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerDied","Data":"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"}
Sep 05 02:57:51.123385 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:57:51.123368    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:58:08.213973 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:08.213936    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerStarted","Data":"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"}
Sep 05 02:58:08.214463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:08.214014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerStarted","Data":"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"}
Sep 05 02:58:08.214463 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:08.214238    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:58:08.233125 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:08.233074    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podStartSLOduration=2.759030834 podStartE2EDuration="23.233061499s" totalImagesPullingTime="16.664268027s" totalInitContainerRuntime="3.809762638s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:58:08.231861765 +0000 UTC m=+2431.871073588" watchObservedRunningTime="2026-09-05 02:58:08.233061499 +0000 UTC m=+2431.872273299"
Sep 05 02:58:09.218766 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:09.218719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:58:09.220088 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:09.220057    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:10.222584 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:10.222542    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:15.228096 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:15.228064    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:58:15.228660 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:15.228631    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:25.229353 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:25.229307    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:35.228809 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:35.228765    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:45.229393 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:45.229352    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:58:55.229219 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:58:55.229175    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 05 02:59:04.541904 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.541873    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542207    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kube-rbac-proxy"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542222    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kube-rbac-proxy"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542238    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kserve-container"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542247    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kserve-container"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542273    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="storage-initializer"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542282    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="storage-initializer"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542367    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kserve-container"
Sep 05 02:59:04.542408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.542380    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8c862f-3ca8-4acf-b532-65b676f782ca" containerName="kube-rbac-proxy"
Sep 05 02:59:04.544699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.544682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.547334 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.547307    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:59:04.547448 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.547337    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 05 02:59:04.555412 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.555391    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:04.600074 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.600052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.600187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.600087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qsv\" (UniqueName: \"kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.600187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.600108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.600187 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.600170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.600313 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.600282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.700773 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.700731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.700870 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.700801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.700870 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.700825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qsv\" (UniqueName: \"kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.700870 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.700845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.701034 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.700870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.701286 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.701259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.701406 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.701387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.701466 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.701442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.703276 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.703256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.708444 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.708422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qsv\" (UniqueName: \"kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:04.856052 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:04.855987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:05.127713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:05.127691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:05.131612 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:59:05.131577    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e0fb14_37a5_4d54_a45c_c521f10f50c1.slice/crio-554a244a398b129b2e72c185e383bdaf1c4c0d29482c44a5096874bff73f7451 WatchSource:0}: Error finding container 554a244a398b129b2e72c185e383bdaf1c4c0d29482c44a5096874bff73f7451: Status 404 returned error can't find the container with id 554a244a398b129b2e72c185e383bdaf1c4c0d29482c44a5096874bff73f7451
Sep 05 02:59:05.229929 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:05.229903    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:59:05.453848 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:05.453812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerStarted","Data":"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"}
Sep 05 02:59:05.454014 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:05.453891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerStarted","Data":"554a244a398b129b2e72c185e383bdaf1c4c0d29482c44a5096874bff73f7451"}
Sep 05 02:59:09.473351 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:09.473318    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerID="d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0" exitCode=0
Sep 05 02:59:09.473713 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:09.473393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerDied","Data":"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"}
Sep 05 02:59:10.481255 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:10.481221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerStarted","Data":"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"}
Sep 05 02:59:10.481652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:10.481284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerStarted","Data":"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"}
Sep 05 02:59:10.481652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:10.481572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:10.481652 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:10.481632    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:10.504994 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:10.504948    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" podStartSLOduration=2.762779965 podStartE2EDuration="6.504929724s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742149759s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:59:10.504791772 +0000 UTC m=+2494.144003598" watchObservedRunningTime="2026-09-05 02:59:10.504929724 +0000 UTC m=+2494.144141526"
Sep 05 02:59:15.173203 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.173171    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:59:15.173565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.173434    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container" containerID="cri-o://22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421" gracePeriod=30
Sep 05 02:59:15.173565 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.173442    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kube-rbac-proxy" containerID="cri-o://12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb" gracePeriod=30
Sep 05 02:59:15.223315 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.223280    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 05 02:59:15.507021 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.506948    2560 generic.go:356] "Generic (PLEG): container finished" podID="b48af8f0-3687-480b-b7e3-9635d6763944" containerID="12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb" exitCode=2
Sep 05 02:59:15.507170 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:15.507022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerDied","Data":"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"}
Sep 05 02:59:16.491575 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:16.491543    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:18.521549 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.521528    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:59:18.522645 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.522622    2560 generic.go:356] "Generic (PLEG): container finished" podID="b48af8f0-3687-480b-b7e3-9635d6763944" containerID="22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421" exitCode=0
Sep 05 02:59:18.522735 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.522697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerDied","Data":"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"}
Sep 05 02:59:18.522830 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.522784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff" event={"ID":"b48af8f0-3687-480b-b7e3-9635d6763944","Type":"ContainerDied","Data":"0f5119a9fbc49d9e698f673f3ab9dbfb11ee2a1dad451ebe0fff62547570c853"}
Sep 05 02:59:18.522888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.522830    2560 scope.go:117] "RemoveContainer" containerID="12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"
Sep 05 02:59:18.530598 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.530584    2560 scope.go:117] "RemoveContainer" containerID="22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"
Sep 05 02:59:18.538862 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.538837    2560 scope.go:117] "RemoveContainer" containerID="a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"
Sep 05 02:59:18.551401 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.551380    2560 scope.go:117] "RemoveContainer" containerID="12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"
Sep 05 02:59:18.551673 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:18.551657    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb\": container with ID starting with 12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb not found: ID does not exist" containerID="12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"
Sep 05 02:59:18.551722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.551682    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb"} err="failed to get container status \"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb\": rpc error: code = NotFound desc = could not find container \"12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb\": container with ID starting with 12d946ef82435f44e0c0d0eea8f57e58921dcbf968e38d3094949677fb2cefbb not found: ID does not exist"
Sep 05 02:59:18.551722 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.551701    2560 scope.go:117] "RemoveContainer" containerID="22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"
Sep 05 02:59:18.552039 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:18.552020    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421\": container with ID starting with 22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421 not found: ID does not exist" containerID="22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"
Sep 05 02:59:18.552087 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.552050    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421"} err="failed to get container status \"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421\": rpc error: code = NotFound desc = could not find container \"22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421\": container with ID starting with 22ce6f3b19e127eef6073ef36f7494635c45e29bec1a1aff98280fb8404f5421 not found: ID does not exist"
Sep 05 02:59:18.552087 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.552069    2560 scope.go:117] "RemoveContainer" containerID="a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"
Sep 05 02:59:18.552320 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:18.552293    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac\": container with ID starting with a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac not found: ID does not exist" containerID="a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"
Sep 05 02:59:18.552376 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.552328    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac"} err="failed to get container status \"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac\": rpc error: code = NotFound desc = could not find container \"a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac\": container with ID starting with a8a8a52c576a6db85fcecc8db68401602b97d69e2d3626b96acba31833546fac not found: ID does not exist"
Sep 05 02:59:18.609789 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.609713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config\") pod \"b48af8f0-3687-480b-b7e3-9635d6763944\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") "
Sep 05 02:59:18.609880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.609792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location\") pod \"b48af8f0-3687-480b-b7e3-9635d6763944\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") "
Sep 05 02:59:18.609944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.609888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9mb4\" (UniqueName: \"kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4\") pod \"b48af8f0-3687-480b-b7e3-9635d6763944\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") "
Sep 05 02:59:18.609944 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.609921    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert\") pod \"b48af8f0-3687-480b-b7e3-9635d6763944\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") "
Sep 05 02:59:18.610044 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.609954    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") pod \"b48af8f0-3687-480b-b7e3-9635d6763944\" (UID: \"b48af8f0-3687-480b-b7e3-9635d6763944\") "
Sep 05 02:59:18.610095 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b48af8f0-3687-480b-b7e3-9635d6763944" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:18.610095 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610061    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b48af8f0-3687-480b-b7e3-9635d6763944" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:59:18.610259 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b48af8f0-3687-480b-b7e3-9635d6763944" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:18.610324 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610310    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:18.610370 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610330    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b48af8f0-3687-480b-b7e3-9635d6763944-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:18.610370 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.610340    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b48af8f0-3687-480b-b7e3-9635d6763944-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:18.611880 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.611859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b48af8f0-3687-480b-b7e3-9635d6763944" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:59:18.611980 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.611929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4" (OuterVolumeSpecName: "kube-api-access-g9mb4") pod "b48af8f0-3687-480b-b7e3-9635d6763944" (UID: "b48af8f0-3687-480b-b7e3-9635d6763944"). InnerVolumeSpecName "kube-api-access-g9mb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:59:18.711275 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.711251    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9mb4\" (UniqueName: \"kubernetes.io/projected/b48af8f0-3687-480b-b7e3-9635d6763944-kube-api-access-g9mb4\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:18.711275 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:18.711272    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b48af8f0-3687-480b-b7e3-9635d6763944-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:19.528253 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:19.528225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"
Sep 05 02:59:19.548865 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:19.548840    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:59:19.552542 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:19.552511    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fkvff"]
Sep 05 02:59:20.330059 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330028    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 02:59:20.330345 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330333    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kube-rbac-proxy"
Sep 05 02:59:20.330408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330347    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kube-rbac-proxy"
Sep 05 02:59:20.330408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330361    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container"
Sep 05 02:59:20.330408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330367    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container"
Sep 05 02:59:20.330408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330385    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="storage-initializer"
Sep 05 02:59:20.330408 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330390    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="storage-initializer"
Sep 05 02:59:20.330656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330438    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kserve-container"
Sep 05 02:59:20.330656 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.330445    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" containerName="kube-rbac-proxy"
Sep 05 02:59:20.334819 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.334805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.337533 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.337515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 05 02:59:20.337649 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.337538    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:59:20.347078 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.347058    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 02:59:20.424025 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.423998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f429c\" (UniqueName: \"kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.424147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.424053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.424147 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.424123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.424276 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.424187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.424276 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.424257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525307 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f429c\" (UniqueName: \"kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525441 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.525783 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.525734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.526170 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.526145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.526264 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.526231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.528061 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.528033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.533524 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.533503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f429c\" (UniqueName: \"kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.644130 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.644109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:20.916678 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.916649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 02:59:20.918324 ip-10-0-140-13 kubenswrapper[2560]: W0905 02:59:20.918296    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2828b46b_00c4_4176_bffc_dc34cb71827b.slice/crio-ae99846a611595bb2222bf68202d9e57904e7291dd5bfb2b3795aa38239e5e62 WatchSource:0}: Error finding container ae99846a611595bb2222bf68202d9e57904e7291dd5bfb2b3795aa38239e5e62: Status 404 returned error can't find the container with id ae99846a611595bb2222bf68202d9e57904e7291dd5bfb2b3795aa38239e5e62
Sep 05 02:59:20.970228 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:20.970203    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48af8f0-3687-480b-b7e3-9635d6763944" path="/var/lib/kubelet/pods/b48af8f0-3687-480b-b7e3-9635d6763944/volumes"
Sep 05 02:59:21.540817 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:21.540782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerStarted","Data":"33217263711a5df09e228c2cfb3e6f812800e461a98af47ed863e979c5560b20"}
Sep 05 02:59:21.541210 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:21.540856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerStarted","Data":"ae99846a611595bb2222bf68202d9e57904e7291dd5bfb2b3795aa38239e5e62"}
Sep 05 02:59:25.561690 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:25.561655    2560 generic.go:356] "Generic (PLEG): container finished" podID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerID="33217263711a5df09e228c2cfb3e6f812800e461a98af47ed863e979c5560b20" exitCode=0
Sep 05 02:59:25.562122 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:25.561739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerDied","Data":"33217263711a5df09e228c2cfb3e6f812800e461a98af47ed863e979c5560b20"}
Sep 05 02:59:26.568759 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:26.568710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerStarted","Data":"a7e69353df6d00a51af27dfc104d8956f80c0740affd8c9df0f54a25ce65f5dc"}
Sep 05 02:59:26.569177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:26.568784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerStarted","Data":"e187e93ad15d980c2103b8f414d51db39ee514fcbda2003e73ab3fb1ee0030f0"}
Sep 05 02:59:26.569177 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:26.568971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:26.590864 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:26.590824    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podStartSLOduration=2.812658564 podStartE2EDuration="6.59080909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778150526s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:59:26.589585792 +0000 UTC m=+2510.228797617" watchObservedRunningTime="2026-09-05 02:59:26.59080909 +0000 UTC m=+2510.230020891"
Sep 05 02:59:27.574323 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:27.574293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:33.586972 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:33.586900    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 02:59:46.494914 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:46.494884    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:54.576280 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:54.576238    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:54.576724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:54.576496    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kserve-container" containerID="cri-o://bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25" gracePeriod=30
Sep 05 02:59:54.576724 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:54.576555    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kube-rbac-proxy" containerID="cri-o://ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df" gracePeriod=30
Sep 05 02:59:54.700127 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:54.700100    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerID="ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df" exitCode=2
Sep 05 02:59:54.700269 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:54.700174    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerDied","Data":"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"}
Sep 05 02:59:55.520727 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.520707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:55.620217 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2qsv\" (UniqueName: \"kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv\") pod \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") "
Sep 05 02:59:55.620611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620224    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config\") pod \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") "
Sep 05 02:59:55.620611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls\") pod \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") "
Sep 05 02:59:55.620611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert\") pod \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") "
Sep 05 02:59:55.620611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location\") pod \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\" (UID: \"a3e0fb14-37a5-4d54-a45c-c521f10f50c1\") "
Sep 05 02:59:55.620611 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3e0fb14-37a5-4d54-a45c-c521f10f50c1" (UID: "a3e0fb14-37a5-4d54-a45c-c521f10f50c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:55.620861 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3e0fb14-37a5-4d54-a45c-c521f10f50c1" (UID: "a3e0fb14-37a5-4d54-a45c-c521f10f50c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:55.620861 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.620721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3e0fb14-37a5-4d54-a45c-c521f10f50c1" (UID: "a3e0fb14-37a5-4d54-a45c-c521f10f50c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:59:55.622252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.622221    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv" (OuterVolumeSpecName: "kube-api-access-k2qsv") pod "a3e0fb14-37a5-4d54-a45c-c521f10f50c1" (UID: "a3e0fb14-37a5-4d54-a45c-c521f10f50c1"). InnerVolumeSpecName "kube-api-access-k2qsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:59:55.622252 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.622226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3e0fb14-37a5-4d54-a45c-c521f10f50c1" (UID: "a3e0fb14-37a5-4d54-a45c-c521f10f50c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:59:55.706191 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.706161    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerID="bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25" exitCode=0
Sep 05 02:59:55.706310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.706208    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"
Sep 05 02:59:55.706310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.706225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerDied","Data":"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"}
Sep 05 02:59:55.706310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.706273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w" event={"ID":"a3e0fb14-37a5-4d54-a45c-c521f10f50c1","Type":"ContainerDied","Data":"554a244a398b129b2e72c185e383bdaf1c4c0d29482c44a5096874bff73f7451"}
Sep 05 02:59:55.706310 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.706300    2560 scope.go:117] "RemoveContainer" containerID="ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"
Sep 05 02:59:55.714699 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.714684    2560 scope.go:117] "RemoveContainer" containerID="bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"
Sep 05 02:59:55.721276 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.721256    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qsv\" (UniqueName: \"kubernetes.io/projected/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-api-access-k2qsv\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:55.721364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.721278    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:55.721364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.721287    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:55.721364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.721296    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:55.721364 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.721303    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e0fb14-37a5-4d54-a45c-c521f10f50c1-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 02:59:55.722088 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.722076    2560 scope.go:117] "RemoveContainer" containerID="d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"
Sep 05 02:59:55.727928 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.727905    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:55.729532 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.729513    2560 scope.go:117] "RemoveContainer" containerID="ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"
Sep 05 02:59:55.729900 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:55.729872    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df\": container with ID starting with ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df not found: ID does not exist" containerID="ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"
Sep 05 02:59:55.729994 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.729907    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df"} err="failed to get container status \"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df\": rpc error: code = NotFound desc = could not find container \"ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df\": container with ID starting with ae51933334e7728fc8f4b51af4941b474d6006da19f550394d9160fe4c1197df not found: ID does not exist"
Sep 05 02:59:55.729994 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.729926    2560 scope.go:117] "RemoveContainer" containerID="bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"
Sep 05 02:59:55.730257 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:55.730229    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25\": container with ID starting with bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25 not found: ID does not exist" containerID="bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"
Sep 05 02:59:55.730319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.730266    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25"} err="failed to get container status \"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25\": rpc error: code = NotFound desc = could not find container \"bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25\": container with ID starting with bc935efc646b6b714b5ffc37795442abc1df8c26405d98efdede5961fe828b25 not found: ID does not exist"
Sep 05 02:59:55.730319 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.730285    2560 scope.go:117] "RemoveContainer" containerID="d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"
Sep 05 02:59:55.730573 ip-10-0-140-13 kubenswrapper[2560]: E0905 02:59:55.730553    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0\": container with ID starting with d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0 not found: ID does not exist" containerID="d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"
Sep 05 02:59:55.730616 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.730578    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0"} err="failed to get container status \"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0\": rpc error: code = NotFound desc = could not find container \"d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0\": container with ID starting with d04817e0d790bd69d6ab31fdab0016ca9b16f0f672c41157d51e4cadaa228ff0 not found: ID does not exist"
Sep 05 02:59:55.731888 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:55.731870    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-xgk8w"]
Sep 05 02:59:56.969802 ip-10-0-140-13 kubenswrapper[2560]: I0905 02:59:56.969769    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" path="/var/lib/kubelet/pods/a3e0fb14-37a5-4d54-a45c-c521f10f50c1/volumes"
Sep 05 03:00:03.632574 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:03.632532    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 05 03:00:13.590051 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:13.590022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 03:00:20.415228 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:20.415191    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 03:00:20.415766 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:20.415444    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container" containerID="cri-o://e187e93ad15d980c2103b8f414d51db39ee514fcbda2003e73ab3fb1ee0030f0" gracePeriod=30
Sep 05 03:00:20.415766 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:20.415488    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kube-rbac-proxy" containerID="cri-o://a7e69353df6d00a51af27dfc104d8956f80c0740affd8c9df0f54a25ce65f5dc" gracePeriod=30
Sep 05 03:00:20.827206 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:20.827118    2560 generic.go:356] "Generic (PLEG): container finished" podID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerID="a7e69353df6d00a51af27dfc104d8956f80c0740affd8c9df0f54a25ce65f5dc" exitCode=2
Sep 05 03:00:20.827206 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:20.827188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerDied","Data":"a7e69353df6d00a51af27dfc104d8956f80c0740affd8c9df0f54a25ce65f5dc"}
Sep 05 03:00:23.581674 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:23.581629    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 05 03:00:23.588052 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:23.588028    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.60:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 05 03:00:25.030684 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.030644    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:00:25.031149 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031133    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kube-rbac-proxy"
Sep 05 03:00:25.031194 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031152    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kube-rbac-proxy"
Sep 05 03:00:25.031194 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031174    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="storage-initializer"
Sep 05 03:00:25.031194 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031183    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="storage-initializer"
Sep 05 03:00:25.031293 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kserve-container"
Sep 05 03:00:25.031293 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031204    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kserve-container"
Sep 05 03:00:25.031293 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031280    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kube-rbac-proxy"
Sep 05 03:00:25.031384 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.031292    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e0fb14-37a5-4d54-a45c-c521f10f50c1" containerName="kserve-container"
Sep 05 03:00:25.034882 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.034864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.041825 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.041804    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 05 03:00:25.042240 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.042224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 05 03:00:25.065549 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.065524    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:00:25.142318 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.142292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.142447 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.142332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.142447 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.142366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.142530 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.142450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mcq\" (UniqueName: \"kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.142530 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.142504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243445 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243580 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243580 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243580 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243580 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mcq\" (UniqueName: \"kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.243958 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.243933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.244190 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.244170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.244265 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.244249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.245864 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.245848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.250796 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.250779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mcq\" (UniqueName: \"kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.344528 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.344437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:25.615911 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.615889    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:00:25.618319 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:00:25.618288    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db84ec3_8661_4265_8e4f_661aa144aa60.slice/crio-92da9c1f108651e099e25054caed47d3771ee66ff0737516596bc52475edcd5b WatchSource:0}: Error finding container 92da9c1f108651e099e25054caed47d3771ee66ff0737516596bc52475edcd5b: Status 404 returned error can't find the container with id 92da9c1f108651e099e25054caed47d3771ee66ff0737516596bc52475edcd5b
Sep 05 03:00:25.854987 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.854948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerStarted","Data":"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"}
Sep 05 03:00:25.855143 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:25.855029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerStarted","Data":"92da9c1f108651e099e25054caed47d3771ee66ff0737516596bc52475edcd5b"}
Sep 05 03:00:26.862213 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:26.862181    2560 generic.go:356] "Generic (PLEG): container finished" podID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerID="e187e93ad15d980c2103b8f414d51db39ee514fcbda2003e73ab3fb1ee0030f0" exitCode=0
Sep 05 03:00:26.862533 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:26.862254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerDied","Data":"e187e93ad15d980c2103b8f414d51db39ee514fcbda2003e73ab3fb1ee0030f0"}
Sep 05 03:00:26.952253 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:26.952229    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 03:00:27.058422 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location\") pod \"2828b46b-00c4-4176-bffc-dc34cb71827b\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") "
Sep 05 03:00:27.058422 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058389    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert\") pod \"2828b46b-00c4-4176-bffc-dc34cb71827b\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") "
Sep 05 03:00:27.058422 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058421    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f429c\" (UniqueName: \"kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c\") pod \"2828b46b-00c4-4176-bffc-dc34cb71827b\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") "
Sep 05 03:00:27.058667 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058459    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls\") pod \"2828b46b-00c4-4176-bffc-dc34cb71827b\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") "
Sep 05 03:00:27.058667 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config\") pod \"2828b46b-00c4-4176-bffc-dc34cb71827b\" (UID: \"2828b46b-00c4-4176-bffc-dc34cb71827b\") "
Sep 05 03:00:27.058809 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058672    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2828b46b-00c4-4176-bffc-dc34cb71827b" (UID: "2828b46b-00c4-4176-bffc-dc34cb71827b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:27.058809 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058781    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2828b46b-00c4-4176-bffc-dc34cb71827b" (UID: "2828b46b-00c4-4176-bffc-dc34cb71827b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:00:27.059187 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.058978    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2828b46b-00c4-4176-bffc-dc34cb71827b" (UID: "2828b46b-00c4-4176-bffc-dc34cb71827b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:00:27.059187 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.059016    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2828b46b-00c4-4176-bffc-dc34cb71827b-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:00:27.059385 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.059270    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:00:27.060716 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.060692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c" (OuterVolumeSpecName: "kube-api-access-f429c") pod "2828b46b-00c4-4176-bffc-dc34cb71827b" (UID: "2828b46b-00c4-4176-bffc-dc34cb71827b"). InnerVolumeSpecName "kube-api-access-f429c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:27.060924 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.060760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2828b46b-00c4-4176-bffc-dc34cb71827b" (UID: "2828b46b-00c4-4176-bffc-dc34cb71827b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:27.159912 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.159885    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:00:27.159912 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.159909    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f429c\" (UniqueName: \"kubernetes.io/projected/2828b46b-00c4-4176-bffc-dc34cb71827b-kube-api-access-f429c\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:00:27.160055 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.159924    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2828b46b-00c4-4176-bffc-dc34cb71827b-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:00:27.869306 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.869277    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"
Sep 05 03:00:27.869721 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.869269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb" event={"ID":"2828b46b-00c4-4176-bffc-dc34cb71827b","Type":"ContainerDied","Data":"ae99846a611595bb2222bf68202d9e57904e7291dd5bfb2b3795aa38239e5e62"}
Sep 05 03:00:27.869721 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.869442    2560 scope.go:117] "RemoveContainer" containerID="a7e69353df6d00a51af27dfc104d8956f80c0740affd8c9df0f54a25ce65f5dc"
Sep 05 03:00:27.878548 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.878526    2560 scope.go:117] "RemoveContainer" containerID="e187e93ad15d980c2103b8f414d51db39ee514fcbda2003e73ab3fb1ee0030f0"
Sep 05 03:00:27.886100 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.886084    2560 scope.go:117] "RemoveContainer" containerID="33217263711a5df09e228c2cfb3e6f812800e461a98af47ed863e979c5560b20"
Sep 05 03:00:27.891556 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.891537    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 03:00:27.894815 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:27.894793    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zlscb"]
Sep 05 03:00:28.969469 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:28.969434    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" path="/var/lib/kubelet/pods/2828b46b-00c4-4176-bffc-dc34cb71827b/volumes"
Sep 05 03:00:29.882484 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:29.882453    2560 generic.go:356] "Generic (PLEG): container finished" podID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerID="1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1" exitCode=0
Sep 05 03:00:29.882609 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:29.882519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerDied","Data":"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"}
Sep 05 03:00:30.890662 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:30.890626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerStarted","Data":"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"}
Sep 05 03:00:30.891107 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:30.890687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerStarted","Data":"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"}
Sep 05 03:00:30.891107 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:30.890904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:30.891107 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:30.890956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:00:30.915432 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:30.915384    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" podStartSLOduration=1.7893090310000002 podStartE2EDuration="5.915369711s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.12606068s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:00:30.913983638 +0000 UTC m=+2574.553195456" watchObservedRunningTime="2026-09-05 03:00:30.915369711 +0000 UTC m=+2574.554581511"
Sep 05 03:00:36.901177 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:00:36.901143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:01:06.905560 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:06.905481    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:01:15.086586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:15.086545    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:01:15.087088 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:15.086776    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kserve-container" containerID="cri-o://a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c" gracePeriod=30
Sep 05 03:01:15.087088 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:15.086837    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kube-rbac-proxy" containerID="cri-o://ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5" gracePeriod=30
Sep 05 03:01:16.101868 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.101830    2560 generic.go:356] "Generic (PLEG): container finished" podID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerID="ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5" exitCode=2
Sep 05 03:01:16.102216 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.101900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerDied","Data":"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"}
Sep 05 03:01:16.229182 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.229158    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:01:16.326390 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config\") pod \"6db84ec3-8661-4265-8e4f-661aa144aa60\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") "
Sep 05 03:01:16.326390 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326357    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location\") pod \"6db84ec3-8661-4265-8e4f-661aa144aa60\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") "
Sep 05 03:01:16.326586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6mcq\" (UniqueName: \"kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq\") pod \"6db84ec3-8661-4265-8e4f-661aa144aa60\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") "
Sep 05 03:01:16.326586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls\") pod \"6db84ec3-8661-4265-8e4f-661aa144aa60\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") "
Sep 05 03:01:16.326586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326498    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert\") pod \"6db84ec3-8661-4265-8e4f-661aa144aa60\" (UID: \"6db84ec3-8661-4265-8e4f-661aa144aa60\") "
Sep 05 03:01:16.326736 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326695    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6db84ec3-8661-4265-8e4f-661aa144aa60" (UID: "6db84ec3-8661-4265-8e4f-661aa144aa60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:01:16.326736 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326711    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6db84ec3-8661-4265-8e4f-661aa144aa60" (UID: "6db84ec3-8661-4265-8e4f-661aa144aa60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:16.326961 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.326938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6db84ec3-8661-4265-8e4f-661aa144aa60" (UID: "6db84ec3-8661-4265-8e4f-661aa144aa60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:16.328447 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.328426    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6db84ec3-8661-4265-8e4f-661aa144aa60" (UID: "6db84ec3-8661-4265-8e4f-661aa144aa60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:01:16.328552 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.328479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq" (OuterVolumeSpecName: "kube-api-access-p6mcq") pod "6db84ec3-8661-4265-8e4f-661aa144aa60" (UID: "6db84ec3-8661-4265-8e4f-661aa144aa60"). InnerVolumeSpecName "kube-api-access-p6mcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:01:16.427077 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.427044    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:01:16.427077 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.427075    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:01:16.427077 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.427086    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db84ec3-8661-4265-8e4f-661aa144aa60-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:01:16.427259 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.427094    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mcq\" (UniqueName: \"kubernetes.io/projected/6db84ec3-8661-4265-8e4f-661aa144aa60-kube-api-access-p6mcq\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:01:16.427259 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:16.427103    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db84ec3-8661-4265-8e4f-661aa144aa60-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:01:17.107845 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.107814    2560 generic.go:356] "Generic (PLEG): container finished" podID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerID="a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c" exitCode=0
Sep 05 03:01:17.108239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.107867    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"
Sep 05 03:01:17.108239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.107887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerDied","Data":"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"}
Sep 05 03:01:17.108239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.107941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4" event={"ID":"6db84ec3-8661-4265-8e4f-661aa144aa60","Type":"ContainerDied","Data":"92da9c1f108651e099e25054caed47d3771ee66ff0737516596bc52475edcd5b"}
Sep 05 03:01:17.108239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.107975    2560 scope.go:117] "RemoveContainer" containerID="ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"
Sep 05 03:01:17.116546 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.116530    2560 scope.go:117] "RemoveContainer" containerID="a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"
Sep 05 03:01:17.123948 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.123931    2560 scope.go:117] "RemoveContainer" containerID="1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"
Sep 05 03:01:17.126981 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.126960    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:01:17.131205 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.131185    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wpqp4"]
Sep 05 03:01:17.131799 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.131786    2560 scope.go:117] "RemoveContainer" containerID="ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"
Sep 05 03:01:17.132058 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:01:17.132038    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5\": container with ID starting with ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5 not found: ID does not exist" containerID="ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"
Sep 05 03:01:17.132126 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.132066    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5"} err="failed to get container status \"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5\": rpc error: code = NotFound desc = could not find container \"ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5\": container with ID starting with ce77fb38d8c3c9fa4a2aa9faf6e81da0b0ab6f93e559a5c195407741c51ab2c5 not found: ID does not exist"
Sep 05 03:01:17.132126 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.132081    2560 scope.go:117] "RemoveContainer" containerID="a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"
Sep 05 03:01:17.132306 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:01:17.132290    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c\": container with ID starting with a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c not found: ID does not exist" containerID="a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"
Sep 05 03:01:17.132352 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.132312    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c"} err="failed to get container status \"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c\": rpc error: code = NotFound desc = could not find container \"a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c\": container with ID starting with a41e0035cfb090beea250c94f2e27d162b03f0540e56c8c8600ea77fc389778c not found: ID does not exist"
Sep 05 03:01:17.132352 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.132326    2560 scope.go:117] "RemoveContainer" containerID="1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"
Sep 05 03:01:17.132561 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:01:17.132546    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1\": container with ID starting with 1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1 not found: ID does not exist" containerID="1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"
Sep 05 03:01:17.132611 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:17.132563    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1"} err="failed to get container status \"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1\": rpc error: code = NotFound desc = could not find container \"1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1\": container with ID starting with 1ca926dc941a1836af9c2442a43c2895c0fa6e04f536a5d6a6ab5e7320b69cf1 not found: ID does not exist"
Sep 05 03:01:18.970366 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:01:18.970329    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" path="/var/lib/kubelet/pods/6db84ec3-8661-4265-8e4f-661aa144aa60/volumes"
Sep 05 03:02:37.002420 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:37.002347    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 03:02:37.005044 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:37.005023    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 03:02:45.538712 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.538682    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539032    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="storage-initializer"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539044    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="storage-initializer"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539057    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="storage-initializer"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539062    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="storage-initializer"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539071    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kserve-container"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539077    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kserve-container"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539086    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539092    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539108    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539113    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539124    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539129    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539182    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539191    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kube-rbac-proxy"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539198    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db84ec3-8661-4265-8e4f-661aa144aa60" containerName="kserve-container"
Sep 05 03:02:45.539353 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.539206    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2828b46b-00c4-4176-bffc-dc34cb71827b" containerName="kserve-container"
Sep 05 03:02:45.542100 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.542082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.545520 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.545492    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:02:45.545520 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.545492    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 05 03:02:45.545703 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.545494    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:02:45.546407 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.546390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:02:45.546462 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.546408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:02:45.546640 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.546619    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:02:45.546790 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.546649    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 05 03:02:45.556488 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.556469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:02:45.584645 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.584624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.584764 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.584657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb2t\" (UniqueName: \"kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.584764 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.584681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.584866 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.584802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.584866 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.584840    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686055 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686165 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686165 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb2t\" (UniqueName: \"kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686165 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686329 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686387 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:02:45.686348    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 05 03:02:45.686447 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:02:45.686407    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls podName:6becea5b-7727-455f-b848-4391b32b3a6c nodeName:}" failed. No retries permitted until 2026-09-05 03:02:46.1863874 +0000 UTC m=+2709.825599193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-vclc6" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 05 03:02:45.686513 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686879 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.686923 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.686860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:45.694933 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:45.694911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb2t\" (UniqueName: \"kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:46.188692 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:46.188656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:46.191114 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:46.191090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vclc6\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:46.452941 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:46.452867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:02:46.723056 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:46.723014    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:02:46.725422 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:02:46.725392    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6becea5b_7727_455f_b848_4391b32b3a6c.slice/crio-79d6bf54bf205f5424b54e8ba6be36027077987a29f89a27b3314156410f93d0 WatchSource:0}: Error finding container 79d6bf54bf205f5424b54e8ba6be36027077987a29f89a27b3314156410f93d0: Status 404 returned error can't find the container with id 79d6bf54bf205f5424b54e8ba6be36027077987a29f89a27b3314156410f93d0
Sep 05 03:02:47.483931 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:47.483897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerStarted","Data":"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"}
Sep 05 03:02:47.484115 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:47.483978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerStarted","Data":"79d6bf54bf205f5424b54e8ba6be36027077987a29f89a27b3314156410f93d0"}
Sep 05 03:02:52.506439 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:52.506402    2560 generic.go:356] "Generic (PLEG): container finished" podID="6becea5b-7727-455f-b848-4391b32b3a6c" containerID="74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7" exitCode=0
Sep 05 03:02:52.506825 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:52.506475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerDied","Data":"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"}
Sep 05 03:02:52.507632 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:02:52.507618    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:03:04.566111 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.566076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerStarted","Data":"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"}
Sep 05 03:03:04.566494 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.566143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerStarted","Data":"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"}
Sep 05 03:03:04.566494 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.566462    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:04.566575 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.566523    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:04.567676 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.567652    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:04.588715 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:04.588673    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podStartSLOduration=3.460375108 podStartE2EDuration="19.588660792s" totalImagesPullingTime="11.090503725s" totalInitContainerRuntime="5.037781959s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:03:04.58725302 +0000 UTC m=+2728.226464822" watchObservedRunningTime="2026-09-05 03:03:04.588660792 +0000 UTC m=+2728.227872592"
Sep 05 03:03:05.570159 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:05.570117    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:10.574605 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:10.574572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:10.575120 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:10.575093    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:20.575005 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:20.574954    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:21.228729 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.228694    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:03:21.231806 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.231787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.234380 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.234357    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 05 03:03:21.234498 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.234424    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 05 03:03:21.234559 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.234518    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 05 03:03:21.242203 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.242183    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:03:21.386203 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.386178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.386320 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.386210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.386320 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.386236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.386320 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.386276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl999\" (UniqueName: \"kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.386320 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.386313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.487636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.487562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.487636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.487599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.487636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.487624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.487903 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.487645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl999\" (UniqueName: \"kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.487903 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.487670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.488149 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.488122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.488356 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.488336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.488403 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.488336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.490026 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.490005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.495707 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.495682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl999\" (UniqueName: \"kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.542647 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.542621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:21.813796 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:21.813769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:03:21.816363 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:03:21.816333    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ca38fd_1694_40a1_9522_bf4a6ce5b88b.slice/crio-c3aafbe6176813c84fe22dfaf2682733f2abd89b2dbb3ee7a3246e45cacd3da9 WatchSource:0}: Error finding container c3aafbe6176813c84fe22dfaf2682733f2abd89b2dbb3ee7a3246e45cacd3da9: Status 404 returned error can't find the container with id c3aafbe6176813c84fe22dfaf2682733f2abd89b2dbb3ee7a3246e45cacd3da9
Sep 05 03:03:22.642150 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:22.642105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerStarted","Data":"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"}
Sep 05 03:03:22.642343 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:22.642214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerStarted","Data":"c3aafbe6176813c84fe22dfaf2682733f2abd89b2dbb3ee7a3246e45cacd3da9"}
Sep 05 03:03:23.649010 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:23.648975    2560 generic.go:356] "Generic (PLEG): container finished" podID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerID="6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797" exitCode=0
Sep 05 03:03:23.649471 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:23.649049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerDied","Data":"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"}
Sep 05 03:03:24.655778 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:24.655722    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerStarted","Data":"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"}
Sep 05 03:03:24.656177 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:24.655797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerStarted","Data":"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"}
Sep 05 03:03:24.656177 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:24.655826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:24.676182 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:24.676134    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podStartSLOduration=2.51751297 podStartE2EDuration="3.676119176s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.158606206s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:03:24.674807111 +0000 UTC m=+2748.314018933" watchObservedRunningTime="2026-09-05 03:03:24.676119176 +0000 UTC m=+2748.315330976"
Sep 05 03:03:25.660418 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:25.660386    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:25.661501 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:25.661472    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:03:26.664951 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:26.664914    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:03:30.575651 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:30.575599    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:31.670273 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:31.670244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:03:31.670812 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:31.670784    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:03:40.575319 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:40.575281    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 05 03:03:41.671206 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:41.671165    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:03:50.575897 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:50.575864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:51.670886 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:51.670849    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:03:56.568857 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:56.568820    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:03:56.569268 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:56.569087    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container" containerID="cri-o://3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93" gracePeriod=30
Sep 05 03:03:56.569268 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:56.569112    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kube-rbac-proxy" containerID="cri-o://6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0" gracePeriod=30
Sep 05 03:03:56.813244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:56.813216    2560 generic.go:356] "Generic (PLEG): container finished" podID="6becea5b-7727-455f-b848-4391b32b3a6c" containerID="6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0" exitCode=2
Sep 05 03:03:56.813367 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:56.813260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerDied","Data":"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"}
Sep 05 03:03:59.010975 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.010954    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:59.091305 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location\") pod \"6becea5b-7727-455f-b848-4391b32b3a6c\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") "
Sep 05 03:03:59.091444 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091331    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") pod \"6becea5b-7727-455f-b848-4391b32b3a6c\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") "
Sep 05 03:03:59.091444 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config\") pod \"6becea5b-7727-455f-b848-4391b32b3a6c\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") "
Sep 05 03:03:59.091444 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091369    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfb2t\" (UniqueName: \"kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t\") pod \"6becea5b-7727-455f-b848-4391b32b3a6c\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") "
Sep 05 03:03:59.091444 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091423    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert\") pod \"6becea5b-7727-455f-b848-4391b32b3a6c\" (UID: \"6becea5b-7727-455f-b848-4391b32b3a6c\") "
Sep 05 03:03:59.091752 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091725    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6becea5b-7727-455f-b848-4391b32b3a6c" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:03:59.091865 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.091818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6becea5b-7727-455f-b848-4391b32b3a6c" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:03:59.093470 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.093449    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6becea5b-7727-455f-b848-4391b32b3a6c" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:03:59.093562 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.093513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t" (OuterVolumeSpecName: "kube-api-access-wfb2t") pod "6becea5b-7727-455f-b848-4391b32b3a6c" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c"). InnerVolumeSpecName "kube-api-access-wfb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:03:59.101443 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.101387    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6becea5b-7727-455f-b848-4391b32b3a6c" (UID: "6becea5b-7727-455f-b848-4391b32b3a6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:59.192009 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.191983    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:03:59.192109 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.192012    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6becea5b-7727-455f-b848-4391b32b3a6c-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:03:59.192109 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.192023    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6becea5b-7727-455f-b848-4391b32b3a6c-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:03:59.192109 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.192033    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6becea5b-7727-455f-b848-4391b32b3a6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:03:59.192109 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.192042    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfb2t\" (UniqueName: \"kubernetes.io/projected/6becea5b-7727-455f-b848-4391b32b3a6c-kube-api-access-wfb2t\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:03:59.829474 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.829433    2560 generic.go:356] "Generic (PLEG): container finished" podID="6becea5b-7727-455f-b848-4391b32b3a6c" containerID="3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93" exitCode=0
Sep 05 03:03:59.829649 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.829500    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"
Sep 05 03:03:59.829649 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.829518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerDied","Data":"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"}
Sep 05 03:03:59.829649 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.829603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6" event={"ID":"6becea5b-7727-455f-b848-4391b32b3a6c","Type":"ContainerDied","Data":"79d6bf54bf205f5424b54e8ba6be36027077987a29f89a27b3314156410f93d0"}
Sep 05 03:03:59.829649 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.829630    2560 scope.go:117] "RemoveContainer" containerID="6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"
Sep 05 03:03:59.839067 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.839050    2560 scope.go:117] "RemoveContainer" containerID="3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"
Sep 05 03:03:59.847005 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.846987    2560 scope.go:117] "RemoveContainer" containerID="74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"
Sep 05 03:03:59.853118 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.853072    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:03:59.858848 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.858826    2560 scope.go:117] "RemoveContainer" containerID="6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"
Sep 05 03:03:59.859158 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:03:59.859138    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0\": container with ID starting with 6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0 not found: ID does not exist" containerID="6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"
Sep 05 03:03:59.859232 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859167    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0"} err="failed to get container status \"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0\": rpc error: code = NotFound desc = could not find container \"6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0\": container with ID starting with 6c113f85df84f4c576a632782f25e51382608ef749251fcda28243c9925dc2d0 not found: ID does not exist"
Sep 05 03:03:59.859232 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859186    2560 scope.go:117] "RemoveContainer" containerID="3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"
Sep 05 03:03:59.859481 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:03:59.859456    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93\": container with ID starting with 3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93 not found: ID does not exist" containerID="3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"
Sep 05 03:03:59.859534 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859489    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93"} err="failed to get container status \"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93\": rpc error: code = NotFound desc = could not find container \"3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93\": container with ID starting with 3921d450dceeeacb33c7e56a91b9c4a659fd2f4cf9e7c133bb6ef400fa0c1a93 not found: ID does not exist"
Sep 05 03:03:59.859534 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859510    2560 scope.go:117] "RemoveContainer" containerID="74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"
Sep 05 03:03:59.859638 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859622    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vclc6"]
Sep 05 03:03:59.859793 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:03:59.859772    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7\": container with ID starting with 74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7 not found: ID does not exist" containerID="74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"
Sep 05 03:03:59.859843 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:03:59.859804    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7"} err="failed to get container status \"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7\": rpc error: code = NotFound desc = could not find container \"74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7\": container with ID starting with 74b087f62cb8a34221c51c42053aa5a5960c3f66fc2dbd38ba06112b9fb209d7 not found: ID does not exist"
Sep 05 03:04:00.969627 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:00.969594    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" path="/var/lib/kubelet/pods/6becea5b-7727-455f-b848-4391b32b3a6c/volumes"
Sep 05 03:04:01.671355 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:01.671319    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:04:11.671103 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:11.671064    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:04:21.671133 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:21.671093    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:04:31.670753 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:31.670702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:04:41.671903 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:41.671872    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:04:51.247382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:51.247352    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:04:51.247912 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:51.247568    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" containerID="cri-o://c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c" gracePeriod=30
Sep 05 03:04:51.247912 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:51.247604    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kube-rbac-proxy" containerID="cri-o://324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6" gracePeriod=30
Sep 05 03:04:51.665220 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:51.665175    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Sep 05 03:04:51.671638 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:51.671611    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 05 03:04:52.059694 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.059611    2560 generic.go:356] "Generic (PLEG): container finished" podID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerID="324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6" exitCode=2
Sep 05 03:04:52.059694 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.059650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerDied","Data":"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"}
Sep 05 03:04:52.378620 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.378544    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:04:52.378996 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.378961    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="storage-initializer"
Sep 05 03:04:52.378996 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.378976    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="storage-initializer"
Sep 05 03:04:52.378996 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.378986    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container"
Sep 05 03:04:52.378996 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.378992    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container"
Sep 05 03:04:52.379151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.379000    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kube-rbac-proxy"
Sep 05 03:04:52.379151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.379005    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kube-rbac-proxy"
Sep 05 03:04:52.379151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.379053    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kserve-container"
Sep 05 03:04:52.379151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.379061    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6becea5b-7727-455f-b848-4391b32b3a6c" containerName="kube-rbac-proxy"
Sep 05 03:04:52.381213 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.381197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.383932 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.383907    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 05 03:04:52.383932 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.383907    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 05 03:04:52.395191 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.395170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:04:52.498000 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.497972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5zb\" (UniqueName: \"kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.498137 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.498023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.498137 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.498044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.498137 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.498069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.599300 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.599276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.599421 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.599305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.599421 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.599332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.599421 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.599365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5zb\" (UniqueName: \"kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.599734 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.599714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.600075 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.600056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.601736 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.601703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.609014 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.608992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5zb\" (UniqueName: \"kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.691470 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.691445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:04:52.984898 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:52.984874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:04:52.986812 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:04:52.986779    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16166d64_f4cc_4acb_8942_f8b884b22593.slice/crio-635d21ebe0e55ab314d47fc94cb0065eeaf1ce2d58413ccbed80188fac58e720 WatchSource:0}: Error finding container 635d21ebe0e55ab314d47fc94cb0065eeaf1ce2d58413ccbed80188fac58e720: Status 404 returned error can't find the container with id 635d21ebe0e55ab314d47fc94cb0065eeaf1ce2d58413ccbed80188fac58e720
Sep 05 03:04:53.066000 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:53.065970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerStarted","Data":"e59b8d51076115388a6ab2afb5910d1faa1f0ccac04556767c40db2d7e1e6287"}
Sep 05 03:04:53.066115 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:53.066030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerStarted","Data":"635d21ebe0e55ab314d47fc94cb0065eeaf1ce2d58413ccbed80188fac58e720"}
Sep 05 03:04:55.587611 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.587587    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:04:55.625238 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config\") pod \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") "
Sep 05 03:04:55.625338 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert\") pod \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") "
Sep 05 03:04:55.625338 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625296    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl999\" (UniqueName: \"kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999\") pod \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") "
Sep 05 03:04:55.625338 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls\") pod \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") "
Sep 05 03:04:55.625338 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625334    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location\") pod \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\" (UID: \"66ca38fd-1694-40a1-9522-bf4a6ce5b88b\") "
Sep 05 03:04:55.625811 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625782    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66ca38fd-1694-40a1-9522-bf4a6ce5b88b" (UID: "66ca38fd-1694-40a1-9522-bf4a6ce5b88b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:04:55.625811 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625799    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66ca38fd-1694-40a1-9522-bf4a6ce5b88b" (UID: "66ca38fd-1694-40a1-9522-bf4a6ce5b88b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:55.625997 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.625820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66ca38fd-1694-40a1-9522-bf4a6ce5b88b" (UID: "66ca38fd-1694-40a1-9522-bf4a6ce5b88b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:04:55.627420 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.627394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999" (OuterVolumeSpecName: "kube-api-access-wl999") pod "66ca38fd-1694-40a1-9522-bf4a6ce5b88b" (UID: "66ca38fd-1694-40a1-9522-bf4a6ce5b88b"). InnerVolumeSpecName "kube-api-access-wl999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:04:55.627510 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.627486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66ca38fd-1694-40a1-9522-bf4a6ce5b88b" (UID: "66ca38fd-1694-40a1-9522-bf4a6ce5b88b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:04:55.726382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.726326    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl999\" (UniqueName: \"kubernetes.io/projected/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-api-access-wl999\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:04:55.726382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.726347    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:04:55.726382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.726359    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:04:55.726382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.726368    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:04:55.726382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:55.726377    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ca38fd-1694-40a1-9522-bf4a6ce5b88b-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:04:56.083586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.083512    2560 generic.go:356] "Generic (PLEG): container finished" podID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerID="c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c" exitCode=0
Sep 05 03:04:56.083586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.083572    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"
Sep 05 03:04:56.083586 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.083578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerDied","Data":"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"}
Sep 05 03:04:56.083821 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.083627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h" event={"ID":"66ca38fd-1694-40a1-9522-bf4a6ce5b88b","Type":"ContainerDied","Data":"c3aafbe6176813c84fe22dfaf2682733f2abd89b2dbb3ee7a3246e45cacd3da9"}
Sep 05 03:04:56.083821 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.083653    2560 scope.go:117] "RemoveContainer" containerID="324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"
Sep 05 03:04:56.092447 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.092429    2560 scope.go:117] "RemoveContainer" containerID="c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"
Sep 05 03:04:56.099517 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.099499    2560 scope.go:117] "RemoveContainer" containerID="6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"
Sep 05 03:04:56.106819 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.106801    2560 scope.go:117] "RemoveContainer" containerID="324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"
Sep 05 03:04:56.107101 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:04:56.107082    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6\": container with ID starting with 324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6 not found: ID does not exist" containerID="324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"
Sep 05 03:04:56.107170 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107109    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6"} err="failed to get container status \"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6\": rpc error: code = NotFound desc = could not find container \"324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6\": container with ID starting with 324183182d5207e7b0271f1a72dd6879b0fdd36fabc36cf4e4e15ff8df761bf6 not found: ID does not exist"
Sep 05 03:04:56.107170 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107127    2560 scope.go:117] "RemoveContainer" containerID="c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"
Sep 05 03:04:56.107381 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:04:56.107365    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c\": container with ID starting with c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c not found: ID does not exist" containerID="c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"
Sep 05 03:04:56.107422 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107389    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c"} err="failed to get container status \"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c\": rpc error: code = NotFound desc = could not find container \"c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c\": container with ID starting with c434df826153fce57a000456da46ce1ad1c41975498a24937685804efdcc4a8c not found: ID does not exist"
Sep 05 03:04:56.107422 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107405    2560 scope.go:117] "RemoveContainer" containerID="6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"
Sep 05 03:04:56.107651 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107632    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:04:56.107651 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:04:56.107641    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797\": container with ID starting with 6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797 not found: ID does not exist" containerID="6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"
Sep 05 03:04:56.107768 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.107664    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797"} err="failed to get container status \"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797\": rpc error: code = NotFound desc = could not find container \"6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797\": container with ID starting with 6e69b5ec35f322794890c3be86eec855dbd6664f9bd0ec62c622b17f5076b797 not found: ID does not exist"
Sep 05 03:04:56.111664 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.111643    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7c89f879f8-5q25h"]
Sep 05 03:04:56.970535 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:56.970502    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" path="/var/lib/kubelet/pods/66ca38fd-1694-40a1-9522-bf4a6ce5b88b/volumes"
Sep 05 03:04:58.095854 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:58.095826    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/0.log"
Sep 05 03:04:58.096223 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:58.095862    2560 generic.go:356] "Generic (PLEG): container finished" podID="16166d64-f4cc-4acb-8942-f8b884b22593" containerID="e59b8d51076115388a6ab2afb5910d1faa1f0ccac04556767c40db2d7e1e6287" exitCode=1
Sep 05 03:04:58.096223 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:58.095891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerDied","Data":"e59b8d51076115388a6ab2afb5910d1faa1f0ccac04556767c40db2d7e1e6287"}
Sep 05 03:04:59.102456 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:59.102428    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/0.log"
Sep 05 03:04:59.102842 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:04:59.102491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerStarted","Data":"b7b29137a27547fa5b6cab0ab8ff9ecb2e29917c4344e4de3761f3c6f78f98d5"}
Sep 05 03:05:02.350524 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:02.350491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:05:02.350921 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:02.350698    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer" containerID="cri-o://b7b29137a27547fa5b6cab0ab8ff9ecb2e29917c4344e4de3761f3c6f78f98d5" gracePeriod=30
Sep 05 03:05:03.480165 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480130    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:05:03.480624 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480605    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container"
Sep 05 03:05:03.480717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480627    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container"
Sep 05 03:05:03.480717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480642    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kube-rbac-proxy"
Sep 05 03:05:03.480717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480657    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kube-rbac-proxy"
Sep 05 03:05:03.480717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480678    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="storage-initializer"
Sep 05 03:05:03.480717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480686    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="storage-initializer"
Sep 05 03:05:03.480994 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480819    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kserve-container"
Sep 05 03:05:03.480994 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.480832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ca38fd-1694-40a1-9522-bf4a6ce5b88b" containerName="kube-rbac-proxy"
Sep 05 03:05:03.483023 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.483001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.485684 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.485664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 05 03:05:03.485919 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.485905    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 05 03:05:03.486178 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.486157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:05:03.498814 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.498793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:05:03.587934 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.587905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9tr\" (UniqueName: \"kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.588036 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.587940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.588036 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.587989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.588036 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.588004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.588036 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.588032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9tr\" (UniqueName: \"kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689609 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689880 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.689977 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.689934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.691800 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.691779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.697915 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.697894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9tr\" (UniqueName: \"kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:03.793016 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:03.792951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:04.073049 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.073022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:05:04.075522 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:05:04.075494    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21b027c_c658_4979_8b32_728bc39a681e.slice/crio-a85a6ebaa2c0ee909f51035bd6c03b9b4a47f5569bee06a15e723ef0223c902d WatchSource:0}: Error finding container a85a6ebaa2c0ee909f51035bd6c03b9b4a47f5569bee06a15e723ef0223c902d: Status 404 returned error can't find the container with id a85a6ebaa2c0ee909f51035bd6c03b9b4a47f5569bee06a15e723ef0223c902d
Sep 05 03:05:04.125954 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.125935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/1.log"
Sep 05 03:05:04.126534 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.126469    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/0.log"
Sep 05 03:05:04.126534 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.126510    2560 generic.go:356] "Generic (PLEG): container finished" podID="16166d64-f4cc-4acb-8942-f8b884b22593" containerID="b7b29137a27547fa5b6cab0ab8ff9ecb2e29917c4344e4de3761f3c6f78f98d5" exitCode=1
Sep 05 03:05:04.126691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.126592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerDied","Data":"b7b29137a27547fa5b6cab0ab8ff9ecb2e29917c4344e4de3761f3c6f78f98d5"}
Sep 05 03:05:04.126691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.126655    2560 scope.go:117] "RemoveContainer" containerID="e59b8d51076115388a6ab2afb5910d1faa1f0ccac04556767c40db2d7e1e6287"
Sep 05 03:05:04.128671 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.128636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerStarted","Data":"a85a6ebaa2c0ee909f51035bd6c03b9b4a47f5569bee06a15e723ef0223c902d"}
Sep 05 03:05:04.181273 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.181246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/1.log"
Sep 05 03:05:04.182541 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.182520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:05:04.294696 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.294668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location\") pod \"16166d64-f4cc-4acb-8942-f8b884b22593\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") "
Sep 05 03:05:04.294879 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.294711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls\") pod \"16166d64-f4cc-4acb-8942-f8b884b22593\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") "
Sep 05 03:05:04.294879 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.294780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp5zb\" (UniqueName: \"kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb\") pod \"16166d64-f4cc-4acb-8942-f8b884b22593\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") "
Sep 05 03:05:04.294879 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.294801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config\") pod \"16166d64-f4cc-4acb-8942-f8b884b22593\" (UID: \"16166d64-f4cc-4acb-8942-f8b884b22593\") "
Sep 05 03:05:04.295052 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.294974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16166d64-f4cc-4acb-8942-f8b884b22593" (UID: "16166d64-f4cc-4acb-8942-f8b884b22593"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:04.295301 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.295269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16166d64-f4cc-4acb-8942-f8b884b22593" (UID: "16166d64-f4cc-4acb-8942-f8b884b22593"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:05:04.296643 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.296626    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16166d64-f4cc-4acb-8942-f8b884b22593" (UID: "16166d64-f4cc-4acb-8942-f8b884b22593"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:05:04.296808 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.296789    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb" (OuterVolumeSpecName: "kube-api-access-vp5zb") pod "16166d64-f4cc-4acb-8942-f8b884b22593" (UID: "16166d64-f4cc-4acb-8942-f8b884b22593"). InnerVolumeSpecName "kube-api-access-vp5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:05:04.396058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.396026    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16166d64-f4cc-4acb-8942-f8b884b22593-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:05:04.396058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.396053    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16166d64-f4cc-4acb-8942-f8b884b22593-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:05:04.396058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.396064    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5zb\" (UniqueName: \"kubernetes.io/projected/16166d64-f4cc-4acb-8942-f8b884b22593-kube-api-access-vp5zb\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:05:04.396282 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:04.396073    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16166d64-f4cc-4acb-8942-f8b884b22593-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:05:05.135684 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.135644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerStarted","Data":"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"}
Sep 05 03:05:05.137335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.137309    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8_16166d64-f4cc-4acb-8942-f8b884b22593/storage-initializer/1.log"
Sep 05 03:05:05.137450 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.137369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8" event={"ID":"16166d64-f4cc-4acb-8942-f8b884b22593","Type":"ContainerDied","Data":"635d21ebe0e55ab314d47fc94cb0065eeaf1ce2d58413ccbed80188fac58e720"}
Sep 05 03:05:05.137450 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.137409    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"
Sep 05 03:05:05.137450 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.137433    2560 scope.go:117] "RemoveContainer" containerID="b7b29137a27547fa5b6cab0ab8ff9ecb2e29917c4344e4de3761f3c6f78f98d5"
Sep 05 03:05:05.181842 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.181811    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:05:05.188930 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:05.188900    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-557bfd6f98-s4jk8"]
Sep 05 03:05:06.145297 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:06.145261    2560 generic.go:356] "Generic (PLEG): container finished" podID="f21b027c-c658-4979-8b32-728bc39a681e" containerID="61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054" exitCode=0
Sep 05 03:05:06.145688 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:06.145342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerDied","Data":"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"}
Sep 05 03:05:06.970576 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:06.970534    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" path="/var/lib/kubelet/pods/16166d64-f4cc-4acb-8942-f8b884b22593/volumes"
Sep 05 03:05:07.152261 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.152222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerStarted","Data":"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"}
Sep 05 03:05:07.152633 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.152288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerStarted","Data":"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"}
Sep 05 03:05:07.152633 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.152316    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:07.152633 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.152364    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:07.153554 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.153525    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:07.176521 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:07.176461    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podStartSLOduration=3.092759088 podStartE2EDuration="4.176440543s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.083681455s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:05:07.174538609 +0000 UTC m=+2850.813750434" watchObservedRunningTime="2026-09-05 03:05:07.176440543 +0000 UTC m=+2850.815652345"
Sep 05 03:05:08.156978 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:08.156940    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:13.161597 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:13.161563    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:05:13.162200 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:13.162174    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:18.049024 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.048982    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:05:18.049374 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049313    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.049374 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049325    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.049374 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049335    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.049374 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049342    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.049508 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049392    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.049508 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.049402    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="16166d64-f4cc-4acb-8942-f8b884b22593" containerName="storage-initializer"
Sep 05 03:05:18.051363 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.051342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.054008 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.053985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 05 03:05:18.054142 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.054027    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:05:18.054142 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.054030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 05 03:05:18.067272 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.067250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:05:18.205437 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.205406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj272\" (UniqueName: \"kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.205567 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.205446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.205567 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.205467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.205567 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.205483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.205567 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.205541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.306877 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.306808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj272\" (UniqueName: \"kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.306877 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.306851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307067 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.306886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307067 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.306912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307067 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.306959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307221 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.307209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307629 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.307608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.307726 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.307638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.309373 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.309356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.316023 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.315994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj272\" (UniqueName: \"kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.361513 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.361491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:18.633303 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:18.633275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:05:18.635575 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:05:18.635543    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192dd236_8f50_4126_a186_a9974f8dd437.slice/crio-08bd8bf54cd4decb9feac13dfe031f5c2059eee25c54ca4b7d671bc3a6f68dae WatchSource:0}: Error finding container 08bd8bf54cd4decb9feac13dfe031f5c2059eee25c54ca4b7d671bc3a6f68dae: Status 404 returned error can't find the container with id 08bd8bf54cd4decb9feac13dfe031f5c2059eee25c54ca4b7d671bc3a6f68dae
Sep 05 03:05:19.214317 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:19.214262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerStarted","Data":"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"}
Sep 05 03:05:19.214761 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:19.214370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerStarted","Data":"08bd8bf54cd4decb9feac13dfe031f5c2059eee25c54ca4b7d671bc3a6f68dae"}
Sep 05 03:05:23.162424 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:23.162386    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:24.240880 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:24.240848    2560 generic.go:356] "Generic (PLEG): container finished" podID="192dd236-8f50-4126-a186-a9974f8dd437" containerID="c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9" exitCode=0
Sep 05 03:05:24.241254 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:24.240919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerDied","Data":"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"}
Sep 05 03:05:25.247470 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.247429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerStarted","Data":"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"}
Sep 05 03:05:25.247881 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.247491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerStarted","Data":"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"}
Sep 05 03:05:25.247881 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.247773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:25.247993 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.247924    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:25.249428 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.249398    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:05:25.271243 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:25.271200    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podStartSLOduration=2.509154176 podStartE2EDuration="7.271186854s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.762032678s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:05:25.270061367 +0000 UTC m=+2868.909273185" watchObservedRunningTime="2026-09-05 03:05:25.271186854 +0000 UTC m=+2868.910398655"
Sep 05 03:05:26.252497 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:26.252452    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:05:31.257429 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:31.257344    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:05:31.257999 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:31.257970    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:05:33.162793 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:33.162734    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:41.258111 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:41.258064    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:05:43.162778 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:43.162720    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:05:51.258723 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:51.258683    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:05:53.162861 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:05:53.162820    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:06:01.258545 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:01.258501    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:06:03.162763 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:03.162711    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:06:11.259380 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:11.259346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:06:13.163123 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:13.163088    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:06:13.965477 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:13.965436    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:06:19.482540 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:19.482509    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:06:19.483095 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:19.482736    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" containerID="cri-o://07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f" gracePeriod=30
Sep 05 03:06:19.483095 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:19.482818    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kube-rbac-proxy" containerID="cri-o://f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e" gracePeriod=30
Sep 05 03:06:20.499617 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:20.499579    2560 generic.go:356] "Generic (PLEG): container finished" podID="192dd236-8f50-4126-a186-a9974f8dd437" containerID="f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e" exitCode=2
Sep 05 03:06:20.499997 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:20.499633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerDied","Data":"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"}
Sep 05 03:06:21.253259 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.253218    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Sep 05 03:06:21.258535 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.258508    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 05 03:06:21.827515 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.827485    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:06:21.928487 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location\") pod \"192dd236-8f50-4126-a186-a9974f8dd437\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") "
Sep 05 03:06:21.928487 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj272\" (UniqueName: \"kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272\") pod \"192dd236-8f50-4126-a186-a9974f8dd437\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") "
Sep 05 03:06:21.928653 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928533    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert\") pod \"192dd236-8f50-4126-a186-a9974f8dd437\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") "
Sep 05 03:06:21.928653 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config\") pod \"192dd236-8f50-4126-a186-a9974f8dd437\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") "
Sep 05 03:06:21.928797 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928702    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls\") pod \"192dd236-8f50-4126-a186-a9974f8dd437\" (UID: \"192dd236-8f50-4126-a186-a9974f8dd437\") "
Sep 05 03:06:21.928909 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928891    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "192dd236-8f50-4126-a186-a9974f8dd437" (UID: "192dd236-8f50-4126-a186-a9974f8dd437"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:06:21.928953 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.928934    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "192dd236-8f50-4126-a186-a9974f8dd437" (UID: "192dd236-8f50-4126-a186-a9974f8dd437"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:06:21.929034 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.929012    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:21.929034 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.929029    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192dd236-8f50-4126-a186-a9974f8dd437-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:21.930684 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.930659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "192dd236-8f50-4126-a186-a9974f8dd437" (UID: "192dd236-8f50-4126-a186-a9974f8dd437"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:06:21.930684 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.930671    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272" (OuterVolumeSpecName: "kube-api-access-gj272") pod "192dd236-8f50-4126-a186-a9974f8dd437" (UID: "192dd236-8f50-4126-a186-a9974f8dd437"). InnerVolumeSpecName "kube-api-access-gj272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:06:21.938030 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:21.938007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "192dd236-8f50-4126-a186-a9974f8dd437" (UID: "192dd236-8f50-4126-a186-a9974f8dd437"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:22.029576 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.029547    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192dd236-8f50-4126-a186-a9974f8dd437-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:22.029576 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.029573    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj272\" (UniqueName: \"kubernetes.io/projected/192dd236-8f50-4126-a186-a9974f8dd437-kube-api-access-gj272\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:22.029705 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.029587    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192dd236-8f50-4126-a186-a9974f8dd437-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:22.511865 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.511831    2560 generic.go:356] "Generic (PLEG): container finished" podID="192dd236-8f50-4126-a186-a9974f8dd437" containerID="07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f" exitCode=0
Sep 05 03:06:22.512015 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.511871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerDied","Data":"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"}
Sep 05 03:06:22.512015 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.511885    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"
Sep 05 03:06:22.512015 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.511924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd" event={"ID":"192dd236-8f50-4126-a186-a9974f8dd437","Type":"ContainerDied","Data":"08bd8bf54cd4decb9feac13dfe031f5c2059eee25c54ca4b7d671bc3a6f68dae"}
Sep 05 03:06:22.512015 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.511951    2560 scope.go:117] "RemoveContainer" containerID="f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"
Sep 05 03:06:22.521079 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.521063    2560 scope.go:117] "RemoveContainer" containerID="07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"
Sep 05 03:06:22.528428 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.528408    2560 scope.go:117] "RemoveContainer" containerID="c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"
Sep 05 03:06:22.535469 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.535453    2560 scope.go:117] "RemoveContainer" containerID="f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"
Sep 05 03:06:22.535718 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:22.535701    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e\": container with ID starting with f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e not found: ID does not exist" containerID="f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"
Sep 05 03:06:22.535783 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.535724    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e"} err="failed to get container status \"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e\": rpc error: code = NotFound desc = could not find container \"f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e\": container with ID starting with f44005e9dd1face2018dd3ba09626d9480eab36835474d181530d05d605df35e not found: ID does not exist"
Sep 05 03:06:22.535783 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.535752    2560 scope.go:117] "RemoveContainer" containerID="07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"
Sep 05 03:06:22.535996 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:22.535979    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f\": container with ID starting with 07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f not found: ID does not exist" containerID="07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"
Sep 05 03:06:22.536048 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.536001    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f"} err="failed to get container status \"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f\": rpc error: code = NotFound desc = could not find container \"07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f\": container with ID starting with 07a6b138cf77a038d9f8a6e992c94e566bda492ec112aad4f5ae411c48a0c01f not found: ID does not exist"
Sep 05 03:06:22.536048 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.536017    2560 scope.go:117] "RemoveContainer" containerID="c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"
Sep 05 03:06:22.536237 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:22.536221    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9\": container with ID starting with c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9 not found: ID does not exist" containerID="c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"
Sep 05 03:06:22.536278 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.536242    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9"} err="failed to get container status \"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9\": rpc error: code = NotFound desc = could not find container \"c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9\": container with ID starting with c81bde962fe0ba32cb8bf04854e2e615346a97aac233764ac2bb96ead3813ea9 not found: ID does not exist"
Sep 05 03:06:22.540556 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.540536    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:06:22.549214 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.545276    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zckcd"]
Sep 05 03:06:22.970398 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:22.970356    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192dd236-8f50-4126-a186-a9974f8dd437" path="/var/lib/kubelet/pods/192dd236-8f50-4126-a186-a9974f8dd437/volumes"
Sep 05 03:06:23.966453 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:23.966423    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:06:24.644928 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.644898    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645201    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645212    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645231    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kube-rbac-proxy"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645236    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kube-rbac-proxy"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645245    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="storage-initializer"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645251    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="storage-initializer"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645306    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kube-rbac-proxy"
Sep 05 03:06:24.645372 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.645316    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="192dd236-8f50-4126-a186-a9974f8dd437" containerName="kserve-container"
Sep 05 03:06:24.649859 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.649840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.652570 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.652547    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 05 03:06:24.652686 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.652554    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:06:24.652756 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.652683    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 05 03:06:24.659479 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.659457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:06:24.747859 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.747833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.747972 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.747866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.747972 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.747898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgl9\" (UniqueName: \"kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.748060 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.748006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.748060 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.748052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848382 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848487 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848487 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848487 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848465    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848659 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgl9\" (UniqueName: \"kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.848860 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.848841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.849144 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.849120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.849199 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.849183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.850921 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.850901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.858316 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.858294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgl9\" (UniqueName: \"kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9\") pod \"isvc-pmml-predictor-8674795f59-rwz8b\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:24.960308 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:24.960251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:25.232121 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:25.232042    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:06:25.235212 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:06:25.235185    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bcc5d9_a957_4985_aac3_65fe47b6c168.slice/crio-141e6b9854ccb2a633ee43d93ca5aec427a6f377cdccc28aec713431c11569dd WatchSource:0}: Error finding container 141e6b9854ccb2a633ee43d93ca5aec427a6f377cdccc28aec713431c11569dd: Status 404 returned error can't find the container with id 141e6b9854ccb2a633ee43d93ca5aec427a6f377cdccc28aec713431c11569dd
Sep 05 03:06:25.528123 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:25.528038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerStarted","Data":"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"}
Sep 05 03:06:25.528123 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:25.528105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerStarted","Data":"141e6b9854ccb2a633ee43d93ca5aec427a6f377cdccc28aec713431c11569dd"}
Sep 05 03:06:29.547845 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:29.547768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerDied","Data":"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"}
Sep 05 03:06:29.548183 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:29.547765    2560 generic.go:356] "Generic (PLEG): container finished" podID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerID="be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27" exitCode=0
Sep 05 03:06:33.572154 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:33.572117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:06:33.572604 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:33.572391    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" containerID="cri-o://f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3" gracePeriod=30
Sep 05 03:06:33.572604 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:33.572400    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kube-rbac-proxy" containerID="cri-o://d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff" gracePeriod=30
Sep 05 03:06:33.965733 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:33.965689    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 05 03:06:34.590303 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:34.590263    2560 generic.go:356] "Generic (PLEG): container finished" podID="f21b027c-c658-4979-8b32-728bc39a681e" containerID="d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff" exitCode=2
Sep 05 03:06:34.590812 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:34.590332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerDied","Data":"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"}
Sep 05 03:06:37.607528 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:37.607493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerStarted","Data":"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"}
Sep 05 03:06:37.607926 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:37.607557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerStarted","Data":"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"}
Sep 05 03:06:37.607926 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:37.607757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:37.632522 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:37.632462    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podStartSLOduration=2.5425991359999998 podStartE2EDuration="13.632444362s" totalImagesPullingTime="7.225032655s" totalInitContainerRuntime="3.864812571s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:06:37.631947753 +0000 UTC m=+2941.271159554" watchObservedRunningTime="2026-09-05 03:06:37.632444362 +0000 UTC m=+2941.271656166"
Sep 05 03:06:38.157916 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.157873    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Sep 05 03:06:38.420947 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.420893    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:06:38.468528 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location\") pod \"f21b027c-c658-4979-8b32-728bc39a681e\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") "
Sep 05 03:06:38.468660 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468538    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert\") pod \"f21b027c-c658-4979-8b32-728bc39a681e\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") "
Sep 05 03:06:38.468660 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468573    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls\") pod \"f21b027c-c658-4979-8b32-728bc39a681e\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") "
Sep 05 03:06:38.468660 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config\") pod \"f21b027c-c658-4979-8b32-728bc39a681e\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") "
Sep 05 03:06:38.468660 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468629    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs9tr\" (UniqueName: \"kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr\") pod \"f21b027c-c658-4979-8b32-728bc39a681e\" (UID: \"f21b027c-c658-4979-8b32-728bc39a681e\") "
Sep 05 03:06:38.468907 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468787    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f21b027c-c658-4979-8b32-728bc39a681e" (UID: "f21b027c-c658-4979-8b32-728bc39a681e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:38.468973 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468915    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21b027c-c658-4979-8b32-728bc39a681e-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:38.468973 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.468939    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f21b027c-c658-4979-8b32-728bc39a681e" (UID: "f21b027c-c658-4979-8b32-728bc39a681e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:06:38.469071 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.469020    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f21b027c-c658-4979-8b32-728bc39a681e" (UID: "f21b027c-c658-4979-8b32-728bc39a681e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:06:38.470637 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.470609    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f21b027c-c658-4979-8b32-728bc39a681e" (UID: "f21b027c-c658-4979-8b32-728bc39a681e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:06:38.470734 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.470676    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr" (OuterVolumeSpecName: "kube-api-access-fs9tr") pod "f21b027c-c658-4979-8b32-728bc39a681e" (UID: "f21b027c-c658-4979-8b32-728bc39a681e"). InnerVolumeSpecName "kube-api-access-fs9tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:06:38.570152 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.570124    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:38.570152 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.570149    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21b027c-c658-4979-8b32-728bc39a681e-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:38.570300 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.570163    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21b027c-c658-4979-8b32-728bc39a681e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:38.570300 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.570176    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9tr\" (UniqueName: \"kubernetes.io/projected/f21b027c-c658-4979-8b32-728bc39a681e-kube-api-access-fs9tr\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:06:38.619838 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.619804    2560 generic.go:356] "Generic (PLEG): container finished" podID="f21b027c-c658-4979-8b32-728bc39a681e" containerID="f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3" exitCode=0
Sep 05 03:06:38.620239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.619889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"
Sep 05 03:06:38.620239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.619899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerDied","Data":"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"}
Sep 05 03:06:38.620239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.620027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7" event={"ID":"f21b027c-c658-4979-8b32-728bc39a681e","Type":"ContainerDied","Data":"a85a6ebaa2c0ee909f51035bd6c03b9b4a47f5569bee06a15e723ef0223c902d"}
Sep 05 03:06:38.620239 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.620077    2560 scope.go:117] "RemoveContainer" containerID="d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"
Sep 05 03:06:38.620503 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.620488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:38.621907 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.621880    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:06:38.628722 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.628708    2560 scope.go:117] "RemoveContainer" containerID="f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"
Sep 05 03:06:38.636359 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.636344    2560 scope.go:117] "RemoveContainer" containerID="61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"
Sep 05 03:06:38.644430 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.644194    2560 scope.go:117] "RemoveContainer" containerID="d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"
Sep 05 03:06:38.644607 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:38.644577    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff\": container with ID starting with d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff not found: ID does not exist" containerID="d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"
Sep 05 03:06:38.644715 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.644614    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff"} err="failed to get container status \"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff\": rpc error: code = NotFound desc = could not find container \"d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff\": container with ID starting with d21d7630deb506adee813ea942c9745b1c7ee4485e0563d0682c025808190fff not found: ID does not exist"
Sep 05 03:06:38.644715 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.644634    2560 scope.go:117] "RemoveContainer" containerID="f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"
Sep 05 03:06:38.644987 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:38.644964    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3\": container with ID starting with f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3 not found: ID does not exist" containerID="f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"
Sep 05 03:06:38.645031 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.644997    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3"} err="failed to get container status \"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3\": rpc error: code = NotFound desc = could not find container \"f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3\": container with ID starting with f19306bef99309727576ec562f4995a95e8adbbae710cfcdcab84d43382030e3 not found: ID does not exist"
Sep 05 03:06:38.645031 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.645020    2560 scope.go:117] "RemoveContainer" containerID="61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"
Sep 05 03:06:38.645200 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.645182    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:06:38.645303 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:38.645285    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054\": container with ID starting with 61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054 not found: ID does not exist" containerID="61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"
Sep 05 03:06:38.645348 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.645309    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054"} err="failed to get container status \"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054\": rpc error: code = NotFound desc = could not find container \"61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054\": container with ID starting with 61ac61c4a2a926e8bcd4202b94545535d1b67cccc31b6ecbdbe709b4c532e054 not found: ID does not exist"
Sep 05 03:06:38.647889 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.647871    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7bcb4b5dfd-kdpd7"]
Sep 05 03:06:38.970318 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:38.970286    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21b027c-c658-4979-8b32-728bc39a681e" path="/var/lib/kubelet/pods/f21b027c-c658-4979-8b32-728bc39a681e/volumes"
Sep 05 03:06:39.625112 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:39.625067    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:06:44.629604 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:44.629578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:06:44.630085 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:44.630057    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:06:45.867899 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.867864    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868212    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kube-rbac-proxy"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868224    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kube-rbac-proxy"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868239    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="storage-initializer"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868245    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="storage-initializer"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868254    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868259    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868306    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kserve-container"
Sep 05 03:06:45.868335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.868320    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21b027c-c658-4979-8b32-728bc39a681e" containerName="kube-rbac-proxy"
Sep 05 03:06:45.872881 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.872863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:45.875418 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.875393    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 05 03:06:45.875727 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.875710    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 05 03:06:45.876058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.876040    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 05 03:06:45.896211 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.896189    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:06:45.929894 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.929865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2wd\" (UniqueName: \"kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:45.929991 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.929940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:45.930034 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.930001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:45.930074 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.930050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:45.930110 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:45.930098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031092 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031204 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2wd\" (UniqueName: \"kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031316 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031316 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031428 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:46.031354    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 05 03:06:46.031428 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:06:46.031423    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls podName:82eff2dd-0b15-4f6b-b196-ebb446f13664 nodeName:}" failed. No retries permitted until 2026-09-05 03:06:46.531404619 +0000 UTC m=+2950.170616412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 05 03:06:46.031688 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031863 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.031932 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.031880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.042244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.042221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2wd\" (UniqueName: \"kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.534765 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.534711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.537116 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.537089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:46.783227 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:46.783197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:47.060613 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:47.060470    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:06:47.063415 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:06:47.063389    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82eff2dd_0b15_4f6b_b196_ebb446f13664.slice/crio-1dc220c175e64c84a8ed63bad5ad2a06ac50544fb210683fae74829f9d4e70af WatchSource:0}: Error finding container 1dc220c175e64c84a8ed63bad5ad2a06ac50544fb210683fae74829f9d4e70af: Status 404 returned error can't find the container with id 1dc220c175e64c84a8ed63bad5ad2a06ac50544fb210683fae74829f9d4e70af
Sep 05 03:06:47.669802 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:47.669765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerStarted","Data":"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"}
Sep 05 03:06:47.670002 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:47.669832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerStarted","Data":"1dc220c175e64c84a8ed63bad5ad2a06ac50544fb210683fae74829f9d4e70af"}
Sep 05 03:06:48.675987 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:48.675950    2560 generic.go:356] "Generic (PLEG): container finished" podID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerID="a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8" exitCode=0
Sep 05 03:06:48.676437 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:48.676020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerDied","Data":"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"}
Sep 05 03:06:49.683084 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:49.683046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerStarted","Data":"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"}
Sep 05 03:06:49.683491 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:49.683102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerStarted","Data":"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"}
Sep 05 03:06:49.683491 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:49.683187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:49.704961 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:49.704917    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podStartSLOduration=3.658676078 podStartE2EDuration="4.704903097s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.046227019s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:06:49.702831345 +0000 UTC m=+2953.342043165" watchObservedRunningTime="2026-09-05 03:06:49.704903097 +0000 UTC m=+2953.344114897"
Sep 05 03:06:50.687400 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:50.687366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:50.688670 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:50.688644    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:06:51.692527 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:51.692481    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:06:54.630636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:54.630597    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:06:56.696904 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:56.696872    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:06:56.697505 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:06:56.697474    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:04.630419 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:04.630334    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:07:06.697707 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:06.697665    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:14.630660 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:14.630616    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:07:16.697471 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:16.697429    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:24.630880 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:24.630822    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:07:26.697986 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:26.697945    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:34.630667 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:34.630625    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:07:36.697578 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:36.697532    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:37.034412 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:37.034344    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 03:07:37.038164 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:37.038143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 03:07:44.630861 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:44.630811    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 05 03:07:46.697649 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:46.697610    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:07:54.630940 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:54.630905    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:07:56.697416 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:07:56.697381    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:08:05.688410 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:05.688372    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:08:05.688890 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:05.688720    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container" containerID="cri-o://82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9" gracePeriod=30
Sep 05 03:08:05.688890 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:05.688770    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kube-rbac-proxy" containerID="cri-o://720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329" gracePeriod=30
Sep 05 03:08:06.048644 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:06.048558    2560 generic.go:356] "Generic (PLEG): container finished" podID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerID="720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329" exitCode=2
Sep 05 03:08:06.048644 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:06.048604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerDied","Data":"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"}
Sep 05 03:08:06.697974 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:06.697940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:08:08.933200 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:08.933176    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:08:09.063127 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.063055    2560 generic.go:356] "Generic (PLEG): container finished" podID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerID="82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9" exitCode=0
Sep 05 03:08:09.063244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.063122    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"
Sep 05 03:08:09.063244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.063145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerDied","Data":"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"}
Sep 05 03:08:09.063244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.063210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b" event={"ID":"f9bcc5d9-a957-4985-aac3-65fe47b6c168","Type":"ContainerDied","Data":"141e6b9854ccb2a633ee43d93ca5aec427a6f377cdccc28aec713431c11569dd"}
Sep 05 03:08:09.063354 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.063251    2560 scope.go:117] "RemoveContainer" containerID="720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"
Sep 05 03:08:09.071388 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.071374    2560 scope.go:117] "RemoveContainer" containerID="82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"
Sep 05 03:08:09.078822 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.078800    2560 scope.go:117] "RemoveContainer" containerID="be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"
Sep 05 03:08:09.086001 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.085984    2560 scope.go:117] "RemoveContainer" containerID="720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"
Sep 05 03:08:09.086275 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:09.086247    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329\": container with ID starting with 720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329 not found: ID does not exist" containerID="720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"
Sep 05 03:08:09.086371 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.086280    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329"} err="failed to get container status \"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329\": rpc error: code = NotFound desc = could not find container \"720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329\": container with ID starting with 720b1f14ef3434dbc1b753670e20b9700faba3333ea9d36d6747fd52bb83b329 not found: ID does not exist"
Sep 05 03:08:09.086371 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.086296    2560 scope.go:117] "RemoveContainer" containerID="82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"
Sep 05 03:08:09.086550 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:09.086526    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9\": container with ID starting with 82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9 not found: ID does not exist" containerID="82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"
Sep 05 03:08:09.086590 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.086557    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9"} err="failed to get container status \"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9\": rpc error: code = NotFound desc = could not find container \"82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9\": container with ID starting with 82b870e0110985ff954cece6072ba88691576c0a88e4c3e188c02707e69641e9 not found: ID does not exist"
Sep 05 03:08:09.086590 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.086572    2560 scope.go:117] "RemoveContainer" containerID="be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"
Sep 05 03:08:09.086839 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:09.086822    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27\": container with ID starting with be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27 not found: ID does not exist" containerID="be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"
Sep 05 03:08:09.086906 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.086843    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27"} err="failed to get container status \"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27\": rpc error: code = NotFound desc = could not find container \"be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27\": container with ID starting with be23556c14af89593b9f6a23af143d9b17cdbdf07e63410f646264bab2651f27 not found: ID does not exist"
Sep 05 03:08:09.110140 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert\") pod \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") "
Sep 05 03:08:09.110225 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110158    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config\") pod \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") "
Sep 05 03:08:09.110225 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls\") pod \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") "
Sep 05 03:08:09.110305 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjgl9\" (UniqueName: \"kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9\") pod \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") "
Sep 05 03:08:09.110337 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110314    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location\") pod \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\" (UID: \"f9bcc5d9-a957-4985-aac3-65fe47b6c168\") "
Sep 05 03:08:09.110507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9bcc5d9-a957-4985-aac3-65fe47b6c168" (UID: "f9bcc5d9-a957-4985-aac3-65fe47b6c168"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:08:09.110582 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110499    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9bcc5d9-a957-4985-aac3-65fe47b6c168" (UID: "f9bcc5d9-a957-4985-aac3-65fe47b6c168"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:08:09.110664 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110645    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:09.110728 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110668    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5d9-a957-4985-aac3-65fe47b6c168-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:09.110728 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.110694    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9bcc5d9-a957-4985-aac3-65fe47b6c168" (UID: "f9bcc5d9-a957-4985-aac3-65fe47b6c168"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:08:09.112055 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.112028    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9bcc5d9-a957-4985-aac3-65fe47b6c168" (UID: "f9bcc5d9-a957-4985-aac3-65fe47b6c168"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:08:09.112226 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.112203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9" (OuterVolumeSpecName: "kube-api-access-wjgl9") pod "f9bcc5d9-a957-4985-aac3-65fe47b6c168" (UID: "f9bcc5d9-a957-4985-aac3-65fe47b6c168"). InnerVolumeSpecName "kube-api-access-wjgl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:08:09.211208 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.211176    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgl9\" (UniqueName: \"kubernetes.io/projected/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kube-api-access-wjgl9\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:09.211208 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.211210    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5d9-a957-4985-aac3-65fe47b6c168-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:09.211336 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.211221    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9bcc5d9-a957-4985-aac3-65fe47b6c168-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:09.385682 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.385656    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:08:09.389546 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:09.389526    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rwz8b"]
Sep 05 03:08:10.969876 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:10.969841    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" path="/var/lib/kubelet/pods/f9bcc5d9-a957-4985-aac3-65fe47b6c168/volumes"
Sep 05 03:08:15.759906 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:15.759869    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:08:15.760275 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:15.760105    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" containerID="cri-o://82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66" gracePeriod=30
Sep 05 03:08:15.760275 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:15.760135    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kube-rbac-proxy" containerID="cri-o://162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8" gracePeriod=30
Sep 05 03:08:16.097258 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:16.097172    2560 generic.go:356] "Generic (PLEG): container finished" podID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerID="162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8" exitCode=2
Sep 05 03:08:16.097258 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:16.097214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerDied","Data":"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"}
Sep 05 03:08:16.692990 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:16.692941    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Sep 05 03:08:16.697648 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:16.697619    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 05 03:08:19.898383 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.898357    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:08:19.995313 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995250    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls2wd\" (UniqueName: \"kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd\") pod \"82eff2dd-0b15-4f6b-b196-ebb446f13664\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") "
Sep 05 03:08:19.995313 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location\") pod \"82eff2dd-0b15-4f6b-b196-ebb446f13664\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") "
Sep 05 03:08:19.995461 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config\") pod \"82eff2dd-0b15-4f6b-b196-ebb446f13664\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") "
Sep 05 03:08:19.995461 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995386    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert\") pod \"82eff2dd-0b15-4f6b-b196-ebb446f13664\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") "
Sep 05 03:08:19.995461 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995420    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") pod \"82eff2dd-0b15-4f6b-b196-ebb446f13664\" (UID: \"82eff2dd-0b15-4f6b-b196-ebb446f13664\") "
Sep 05 03:08:19.995627 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995591    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82eff2dd-0b15-4f6b-b196-ebb446f13664" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:08:19.995723 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82eff2dd-0b15-4f6b-b196-ebb446f13664" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:08:19.995828 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.995807    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82eff2dd-0b15-4f6b-b196-ebb446f13664" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:08:19.997274 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.997250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd" (OuterVolumeSpecName: "kube-api-access-ls2wd") pod "82eff2dd-0b15-4f6b-b196-ebb446f13664" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664"). InnerVolumeSpecName "kube-api-access-ls2wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:08:19.997384 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:19.997366    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82eff2dd-0b15-4f6b-b196-ebb446f13664" (UID: "82eff2dd-0b15-4f6b-b196-ebb446f13664"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:08:20.097122 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.097085    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:20.097122 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.097115    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82eff2dd-0b15-4f6b-b196-ebb446f13664-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:20.097324 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.097128    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls2wd\" (UniqueName: \"kubernetes.io/projected/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-api-access-ls2wd\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:20.097324 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.097143    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82eff2dd-0b15-4f6b-b196-ebb446f13664-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:20.097324 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.097161    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82eff2dd-0b15-4f6b-b196-ebb446f13664-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:08:20.116339 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.116305    2560 generic.go:356] "Generic (PLEG): container finished" podID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerID="82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66" exitCode=0
Sep 05 03:08:20.116440 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.116363    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"
Sep 05 03:08:20.116440 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.116391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerDied","Data":"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"}
Sep 05 03:08:20.116511 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.116451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2" event={"ID":"82eff2dd-0b15-4f6b-b196-ebb446f13664","Type":"ContainerDied","Data":"1dc220c175e64c84a8ed63bad5ad2a06ac50544fb210683fae74829f9d4e70af"}
Sep 05 03:08:20.116511 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.116478    2560 scope.go:117] "RemoveContainer" containerID="162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"
Sep 05 03:08:20.125441 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.125426    2560 scope.go:117] "RemoveContainer" containerID="82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"
Sep 05 03:08:20.132915 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.132898    2560 scope.go:117] "RemoveContainer" containerID="a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"
Sep 05 03:08:20.140706 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.140685    2560 scope.go:117] "RemoveContainer" containerID="162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"
Sep 05 03:08:20.140993 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:20.140974    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8\": container with ID starting with 162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8 not found: ID does not exist" containerID="162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"
Sep 05 03:08:20.141058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141007    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8"} err="failed to get container status \"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8\": rpc error: code = NotFound desc = could not find container \"162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8\": container with ID starting with 162c38b3f33e5663e1065ad59e98b9a8a762b75a857fca501a643ef5469ac4a8 not found: ID does not exist"
Sep 05 03:08:20.141058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141028    2560 scope.go:117] "RemoveContainer" containerID="82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"
Sep 05 03:08:20.141195 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141160    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:08:20.141296 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:20.141276    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66\": container with ID starting with 82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66 not found: ID does not exist" containerID="82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"
Sep 05 03:08:20.141356 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141305    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66"} err="failed to get container status \"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66\": rpc error: code = NotFound desc = could not find container \"82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66\": container with ID starting with 82b4c78f47ab39de224b90edb41ad34c83147b626490ae776f91110aaa14ad66 not found: ID does not exist"
Sep 05 03:08:20.141356 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141328    2560 scope.go:117] "RemoveContainer" containerID="a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"
Sep 05 03:08:20.141586 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:08:20.141569    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8\": container with ID starting with a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8 not found: ID does not exist" containerID="a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"
Sep 05 03:08:20.141630 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.141592    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8"} err="failed to get container status \"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8\": rpc error: code = NotFound desc = could not find container \"a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8\": container with ID starting with a35fde63c797cbfe2e48b804cd5d65870d25db0db7862c5c213defa3aaef9af8 not found: ID does not exist"
Sep 05 03:08:20.144385 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.144363    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-57cb7c849c-dh6h2"]
Sep 05 03:08:20.969537 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:20.969493    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" path="/var/lib/kubelet/pods/82eff2dd-0b15-4f6b-b196-ebb446f13664/volumes"
Sep 05 03:08:28.035183 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035453    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035464    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035474    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kube-rbac-proxy"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035479    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kube-rbac-proxy"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035489    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="storage-initializer"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035495    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="storage-initializer"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035501    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kube-rbac-proxy"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035506    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kube-rbac-proxy"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035513    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035518    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="storage-initializer"
Sep 05 03:08:28.035547 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035529    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="storage-initializer"
Sep 05 03:08:28.035959 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035575    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kserve-container"
Sep 05 03:08:28.035959 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035584    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82eff2dd-0b15-4f6b-b196-ebb446f13664" containerName="kube-rbac-proxy"
Sep 05 03:08:28.035959 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035590    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kserve-container"
Sep 05 03:08:28.035959 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.035597    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bcc5d9-a957-4985-aac3-65fe47b6c168" containerName="kube-rbac-proxy"
Sep 05 03:08:28.038857 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.038828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.041701 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.041680    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6920a4-kube-rbac-proxy-sar-config\""
Sep 05 03:08:28.041701 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.041693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:08:28.041892 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.041700    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:08:28.041892 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.041841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:08:28.042597 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.042578    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:08:28.042641 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.042609    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:08:28.043098 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.043079    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6920a4-predictor-serving-cert\""
Sep 05 03:08:28.048430 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.048405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.048702 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.048651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.048835 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.048803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8jl\" (UniqueName: \"kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.048835 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.048832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.048990 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.048930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.050163 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.050143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:08:28.149618 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.149593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8jl\" (UniqueName: \"kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.149789 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.149621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.149789 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.149659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.149789 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.149689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.149789 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.149724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.150058 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.150038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.150462 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.150438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.150537 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.150438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.152064 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.152046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.158332 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.158306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8jl\" (UniqueName: \"kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl\") pod \"isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") " pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.350653 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.350583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:28.623391 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:28.623220    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:08:28.626189 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:08:28.626158    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a444250_d4d5_4fe3_822b_f355b995ca4d.slice/crio-860388b48921197b214c03c9f42b0b1a6ec94f294ec727aa15bbb0d2cf157067 WatchSource:0}: Error finding container 860388b48921197b214c03c9f42b0b1a6ec94f294ec727aa15bbb0d2cf157067: Status 404 returned error can't find the container with id 860388b48921197b214c03c9f42b0b1a6ec94f294ec727aa15bbb0d2cf157067
Sep 05 03:08:29.157502 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:29.157431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerStarted","Data":"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"}
Sep 05 03:08:29.157899 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:29.157507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerStarted","Data":"860388b48921197b214c03c9f42b0b1a6ec94f294ec727aa15bbb0d2cf157067"}
Sep 05 03:08:33.178324 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:33.178288    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerID="4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb" exitCode=0
Sep 05 03:08:33.178812 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:33.178329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerDied","Data":"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"}
Sep 05 03:08:34.185293 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:34.185257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerStarted","Data":"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"}
Sep 05 03:08:34.185634 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:34.185351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerStarted","Data":"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"}
Sep 05 03:08:34.185634 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:34.185556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:34.213007 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:34.212965    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podStartSLOduration=2.240730269 podStartE2EDuration="6.212953191s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972222922s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:08:34.211476689 +0000 UTC m=+3057.850688534" watchObservedRunningTime="2026-09-05 03:08:34.212953191 +0000 UTC m=+3057.852164994"
Sep 05 03:08:35.189689 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:35.189656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:35.191043 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:35.191017    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:08:36.194200 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:36.194157    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:08:41.199973 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:41.199941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:08:41.200562 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:41.200535    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:08:51.200679 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:08:51.200636    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:01.201007 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:01.200966    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:11.200543 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:11.200502    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:21.201119 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:21.201076    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:31.200808 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:31.200765    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:41.200933 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:41.200890    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:09:50.970327 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:09:50.970289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:10:29.252643 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:29.252612    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:10:29.253213 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:29.252921    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" containerID="cri-o://221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f" gracePeriod=30
Sep 05 03:10:29.253213 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:29.252974    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kube-rbac-proxy" containerID="cri-o://059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc" gracePeriod=30
Sep 05 03:10:29.687808 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:29.687777    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerID="059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc" exitCode=2
Sep 05 03:10:29.687972 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:29.687854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerDied","Data":"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"}
Sep 05 03:10:30.966195 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:30.966153    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 05 03:10:31.194856 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:31.194813    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Sep 05 03:10:33.288541 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.288515    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:10:33.408145 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj8jl\" (UniqueName: \"kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl\") pod \"1a444250-d4d5-4fe3-822b-f355b995ca4d\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") "
Sep 05 03:10:33.408271 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408176    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location\") pod \"1a444250-d4d5-4fe3-822b-f355b995ca4d\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") "
Sep 05 03:10:33.408271 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408199    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config\") pod \"1a444250-d4d5-4fe3-822b-f355b995ca4d\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") "
Sep 05 03:10:33.408271 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls\") pod \"1a444250-d4d5-4fe3-822b-f355b995ca4d\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") "
Sep 05 03:10:33.408271 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert\") pod \"1a444250-d4d5-4fe3-822b-f355b995ca4d\" (UID: \"1a444250-d4d5-4fe3-822b-f355b995ca4d\") "
Sep 05 03:10:33.408616 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a444250-d4d5-4fe3-822b-f355b995ca4d" (UID: "1a444250-d4d5-4fe3-822b-f355b995ca4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:33.408726 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a444250-d4d5-4fe3-822b-f355b995ca4d" (UID: "1a444250-d4d5-4fe3-822b-f355b995ca4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:10:33.408726 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.408695    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a444250-d4d5-4fe3-822b-f355b995ca4d" (UID: "1a444250-d4d5-4fe3-822b-f355b995ca4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:10:33.410145 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.410123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl" (OuterVolumeSpecName: "kube-api-access-cj8jl") pod "1a444250-d4d5-4fe3-822b-f355b995ca4d" (UID: "1a444250-d4d5-4fe3-822b-f355b995ca4d"). InnerVolumeSpecName "kube-api-access-cj8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:10:33.410250 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.410153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a444250-d4d5-4fe3-822b-f355b995ca4d" (UID: "1a444250-d4d5-4fe3-822b-f355b995ca4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:10:33.509207 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.509182    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a444250-d4d5-4fe3-822b-f355b995ca4d-kserve-provision-location\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:10:33.509311 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.509209    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:10:33.509311 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.509224    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a444250-d4d5-4fe3-822b-f355b995ca4d-proxy-tls\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:10:33.509311 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.509239    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a444250-d4d5-4fe3-822b-f355b995ca4d-cabundle-cert\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:10:33.509311 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.509251    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8jl\" (UniqueName: \"kubernetes.io/projected/1a444250-d4d5-4fe3-822b-f355b995ca4d-kube-api-access-cj8jl\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:10:33.706370 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.706304    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerID="221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f" exitCode=0
Sep 05 03:10:33.706370 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.706363    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"
Sep 05 03:10:33.706518 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.706384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerDied","Data":"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"}
Sep 05 03:10:33.706518 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.706447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq" event={"ID":"1a444250-d4d5-4fe3-822b-f355b995ca4d","Type":"ContainerDied","Data":"860388b48921197b214c03c9f42b0b1a6ec94f294ec727aa15bbb0d2cf157067"}
Sep 05 03:10:33.706518 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.706475    2560 scope.go:117] "RemoveContainer" containerID="059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"
Sep 05 03:10:33.715518 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.715499    2560 scope.go:117] "RemoveContainer" containerID="221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"
Sep 05 03:10:33.722601 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.722582    2560 scope.go:117] "RemoveContainer" containerID="4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"
Sep 05 03:10:33.729179 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.729159    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:10:33.729844 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.729828    2560 scope.go:117] "RemoveContainer" containerID="059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"
Sep 05 03:10:33.730118 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:10:33.730096    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc\": container with ID starting with 059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc not found: ID does not exist" containerID="059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"
Sep 05 03:10:33.730194 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.730130    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc"} err="failed to get container status \"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc\": rpc error: code = NotFound desc = could not find container \"059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc\": container with ID starting with 059e0b4e395799defd8290d7af062ac6225404a3393f32f728ecde0f4fb5cdcc not found: ID does not exist"
Sep 05 03:10:33.730194 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.730153    2560 scope.go:117] "RemoveContainer" containerID="221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"
Sep 05 03:10:33.730410 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:10:33.730392    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f\": container with ID starting with 221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f not found: ID does not exist" containerID="221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"
Sep 05 03:10:33.730451 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.730418    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f"} err="failed to get container status \"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f\": rpc error: code = NotFound desc = could not find container \"221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f\": container with ID starting with 221e1a321f75e5b846e30554a1b585f9fe0ce5b05ea421911f0dedf8111c289f not found: ID does not exist"
Sep 05 03:10:33.730451 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.730433    2560 scope.go:117] "RemoveContainer" containerID="4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"
Sep 05 03:10:33.730660 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:10:33.730645    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb\": container with ID starting with 4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb not found: ID does not exist" containerID="4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"
Sep 05 03:10:33.730706 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.730667    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb"} err="failed to get container status \"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb\": rpc error: code = NotFound desc = could not find container \"4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb\": container with ID starting with 4e492515054ad3d3017ce5d7677fd283acad6b4729cbf7564888b5f21a384afb not found: ID does not exist"
Sep 05 03:10:33.733343 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:33.733324    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6920a4-predictor-f6d6f9d8b-vszqq"]
Sep 05 03:10:34.970022 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:10:34.969992    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" path="/var/lib/kubelet/pods/1a444250-d4d5-4fe3-822b-f355b995ca4d/volumes"
Sep 05 03:11:33.886264 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886184    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8zzxg/must-gather-vt697"]
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886518    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kube-rbac-proxy"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886529    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kube-rbac-proxy"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886540    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="storage-initializer"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886546    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="storage-initializer"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886554    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886559    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886614    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kube-rbac-proxy"
Sep 05 03:11:33.886691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.886620    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a444250-d4d5-4fe3-822b-f355b995ca4d" containerName="kserve-container"
Sep 05 03:11:33.889569 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.889551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:33.892636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.892611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8zzxg\"/\"default-dockercfg-fzzdq\""
Sep 05 03:11:33.892913 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.892890    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8zzxg\"/\"openshift-service-ca.crt\""
Sep 05 03:11:33.893037 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.892920    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8zzxg\"/\"kube-root-ca.crt\""
Sep 05 03:11:33.905534 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:33.905511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzxg/must-gather-vt697"]
Sep 05 03:11:34.051099 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.051070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8x7\" (UniqueName: \"kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.051360 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.051272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.152298 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.152234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.152298 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.152280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8x7\" (UniqueName: \"kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.152551 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.152531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.160950 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.160928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8x7\" (UniqueName: \"kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7\") pod \"must-gather-vt697\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") " pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.212596 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.212578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:11:34.482636 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.482614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzxg/must-gather-vt697"]
Sep 05 03:11:34.484950 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:11:34.484919    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb860ac9a_70bf_4514_bdf0_399a8c8cd3ec.slice/crio-e3ba3b5d080da4e89b5bb859d3b114511c68de98ef2be13abbf4f84f583e7891 WatchSource:0}: Error finding container e3ba3b5d080da4e89b5bb859d3b114511c68de98ef2be13abbf4f84f583e7891: Status 404 returned error can't find the container with id e3ba3b5d080da4e89b5bb859d3b114511c68de98ef2be13abbf4f84f583e7891
Sep 05 03:11:34.486843 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.486823    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:11:34.963810 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:34.963775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzxg/must-gather-vt697" event={"ID":"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec","Type":"ContainerStarted","Data":"e3ba3b5d080da4e89b5bb859d3b114511c68de98ef2be13abbf4f84f583e7891"}
Sep 05 03:11:40.994056 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:40.994023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzxg/must-gather-vt697" event={"ID":"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec","Type":"ContainerStarted","Data":"7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b"}
Sep 05 03:11:40.994056 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:40.994064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzxg/must-gather-vt697" event={"ID":"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec","Type":"ContainerStarted","Data":"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"}
Sep 05 03:11:41.011642 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:11:41.010362    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8zzxg/must-gather-vt697" podStartSLOduration=1.824782993 podStartE2EDuration="8.010341742s" totalImagesPullingTime="6.185558749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:11:41.009571519 +0000 UTC m=+3244.648783319" watchObservedRunningTime="2026-09-05 03:11:41.010341742 +0000 UTC m=+3244.649553544"
Sep 05 03:12:07.111026 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:07.110989    2560 generic.go:356] "Generic (PLEG): container finished" podID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerID="5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6" exitCode=0
Sep 05 03:12:07.111425 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:07.111039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzxg/must-gather-vt697" event={"ID":"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec","Type":"ContainerDied","Data":"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"}
Sep 05 03:12:07.111425 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:07.111350    2560 scope.go:117] "RemoveContainer" containerID="5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"
Sep 05 03:12:07.939492 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:07.939457    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8zzxg_must-gather-vt697_b860ac9a-70bf-4514-bdf0-399a8c8cd3ec/gather/0.log"
Sep 05 03:12:11.373457 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:11.373349    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8zl8r_fda2fbf9-4539-4dec-afee-46b0f5635648/global-pull-secret-syncer/0.log"
Sep 05 03:12:11.530791 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:11.530724    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bsckk_ac596e9e-cfc7-477a-bf81-6541df54e1fd/konnectivity-agent/0.log"
Sep 05 03:12:11.603335 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:11.603310    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-13.ec2.internal_2cd236a272d64976424037015e768d67/haproxy/0.log"
Sep 05 03:12:13.502222 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.502049    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8zzxg/must-gather-vt697"]
Sep 05 03:12:13.502811 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.502262    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-8zzxg/must-gather-vt697" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="copy" containerID="cri-o://7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b" gracePeriod=2
Sep 05 03:12:13.504193 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.504168    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8zzxg/must-gather-vt697"]
Sep 05 03:12:13.504425 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:13.504400    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vt697\" is forbidden: User \"system:node:ip-10-0-140-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8zzxg\": no relationship found between node 'ip-10-0-140-13.ec2.internal' and this object" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:13.731769 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.731731    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8zzxg_must-gather-vt697_b860ac9a-70bf-4514-bdf0-399a8c8cd3ec/copy/0.log"
Sep 05 03:12:13.733161 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.733143    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:13.735253 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:13.735229    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vt697\" is forbidden: User \"system:node:ip-10-0-140-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8zzxg\": no relationship found between node 'ip-10-0-140-13.ec2.internal' and this object" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:13.802815 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.802738    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh8x7\" (UniqueName: \"kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7\") pod \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") "
Sep 05 03:12:13.802912 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.802851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output\") pod \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\" (UID: \"b860ac9a-70bf-4514-bdf0-399a8c8cd3ec\") "
Sep 05 03:12:13.804196 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.804166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" (UID: "b860ac9a-70bf-4514-bdf0-399a8c8cd3ec"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:13.804779 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.804739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7" (OuterVolumeSpecName: "kube-api-access-vh8x7") pod "b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" (UID: "b860ac9a-70bf-4514-bdf0-399a8c8cd3ec"). InnerVolumeSpecName "kube-api-access-vh8x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:12:13.903322 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.903300    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8x7\" (UniqueName: \"kubernetes.io/projected/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-kube-api-access-vh8x7\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:12:13.903322 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:13.903320    2560 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec-must-gather-output\") on node \"ip-10-0-140-13.ec2.internal\" DevicePath \"\""
Sep 05 03:12:14.141721 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.141704    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8zzxg_must-gather-vt697_b860ac9a-70bf-4514-bdf0-399a8c8cd3ec/copy/0.log"
Sep 05 03:12:14.142098 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.142072    2560 generic.go:356] "Generic (PLEG): container finished" podID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerID="7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b" exitCode=143
Sep 05 03:12:14.142166 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.142121    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:14.142166 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.142147    2560 scope.go:117] "RemoveContainer" containerID="7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b"
Sep 05 03:12:14.144188 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:14.144163    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vt697\" is forbidden: User \"system:node:ip-10-0-140-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8zzxg\": no relationship found between node 'ip-10-0-140-13.ec2.internal' and this object" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:14.151475 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.151457    2560 scope.go:117] "RemoveContainer" containerID="5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"
Sep 05 03:12:14.153755 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:14.153717    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vt697\" is forbidden: User \"system:node:ip-10-0-140-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8zzxg\": no relationship found between node 'ip-10-0-140-13.ec2.internal' and this object" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" pod="openshift-must-gather-8zzxg/must-gather-vt697"
Sep 05 03:12:14.163020 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.162997    2560 scope.go:117] "RemoveContainer" containerID="7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b"
Sep 05 03:12:14.163289 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:14.163269    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b\": container with ID starting with 7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b not found: ID does not exist" containerID="7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b"
Sep 05 03:12:14.163345 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.163299    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b"} err="failed to get container status \"7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b\": rpc error: code = NotFound desc = could not find container \"7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b\": container with ID starting with 7c8a60de8146b0ad1298fdb5f731a77b9e7cf7be2e77bc109bd5fb7fa7d8ac9b not found: ID does not exist"
Sep 05 03:12:14.163345 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.163315    2560 scope.go:117] "RemoveContainer" containerID="5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"
Sep 05 03:12:14.163576 ip-10-0-140-13 kubenswrapper[2560]: E0905 03:12:14.163563    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6\": container with ID starting with 5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6 not found: ID does not exist" containerID="5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"
Sep 05 03:12:14.163614 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.163578    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6"} err="failed to get container status \"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6\": rpc error: code = NotFound desc = could not find container \"5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6\": container with ID starting with 5e62772d6e2ed1aa1a1078b67178913e6c1f7c260313441444b380f946530ba6 not found: ID does not exist"
Sep 05 03:12:14.970029 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:14.969993    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" path="/var/lib/kubelet/pods/b860ac9a-70bf-4514-bdf0-399a8c8cd3ec/volumes"
Sep 05 03:12:15.741575 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:15.741537    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-6k4zm_3d9ece26-4f8b-4ea7-9525-fb474583095e/cluster-monitoring-operator/0.log"
Sep 05 03:12:15.840446 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:15.840413    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b7fb5cbb7-7hrtt_7a0d9bfa-071e-4eca-94cc-5e1fea04a5e8/metrics-server/0.log"
Sep 05 03:12:15.863722 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:15.863644    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-c6s24_6d5c85dc-3cea-4c2f-9533-8592ecbfe73f/monitoring-plugin/0.log"
Sep 05 03:12:15.974595 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:15.974569    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmls_2da229ad-b5f6-4656-82a3-57033b84b982/node-exporter/0.log"
Sep 05 03:12:15.997228 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:15.997187    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmls_2da229ad-b5f6-4656-82a3-57033b84b982/kube-rbac-proxy/0.log"
Sep 05 03:12:16.031474 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.031455    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmls_2da229ad-b5f6-4656-82a3-57033b84b982/init-textfile/0.log"
Sep 05 03:12:16.139851 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.139817    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vmmsf_ec34bf33-7aca-414b-8d88-5cdd084b047d/kube-rbac-proxy-main/0.log"
Sep 05 03:12:16.162298 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.162268    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vmmsf_ec34bf33-7aca-414b-8d88-5cdd084b047d/kube-rbac-proxy-self/0.log"
Sep 05 03:12:16.185394 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.185366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vmmsf_ec34bf33-7aca-414b-8d88-5cdd084b047d/openshift-state-metrics/0.log"
Sep 05 03:12:16.227760 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.227722    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/prometheus/0.log"
Sep 05 03:12:16.244420 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.244403    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/config-reloader/0.log"
Sep 05 03:12:16.278363 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.278347    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/thanos-sidecar/0.log"
Sep 05 03:12:16.301027 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.301000    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/kube-rbac-proxy-web/0.log"
Sep 05 03:12:16.324789 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.324756    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/kube-rbac-proxy/0.log"
Sep 05 03:12:16.347244 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.347227    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/kube-rbac-proxy-thanos/0.log"
Sep 05 03:12:16.397144 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.397084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1b88099-048d-4ad6-8cf6-91b5a417031b/init-config-reloader/0.log"
Sep 05 03:12:16.463988 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.463967    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8qvbm_8bceb5c2-cca1-49e0-91f0-9739f956bf2a/prometheus-operator-admission-webhook/0.log"
Sep 05 03:12:16.564964 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.564945    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/thanos-query/0.log"
Sep 05 03:12:16.585403 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.585383    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/kube-rbac-proxy-web/0.log"
Sep 05 03:12:16.606243 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.606221    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/kube-rbac-proxy/0.log"
Sep 05 03:12:16.627540 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.627525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/prom-label-proxy/0.log"
Sep 05 03:12:16.652040 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.651977    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/kube-rbac-proxy-rules/0.log"
Sep 05 03:12:16.674379 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:16.674360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-544c549dfd-rsz54_df6028fe-8e6d-4c74-9d01-b66f396b668c/kube-rbac-proxy-metrics/0.log"
Sep 05 03:12:18.950076 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:18.950043    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59fd47449d-q5dcq_124dd00b-9787-4a17-aa02-142fce353e8a/console/0.log"
Sep 05 03:12:19.323466 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323390    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"]
Sep 05 03:12:19.323691 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323679    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="copy"
Sep 05 03:12:19.323731 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323693    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="copy"
Sep 05 03:12:19.323731 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323708    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="gather"
Sep 05 03:12:19.323731 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323714    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="gather"
Sep 05 03:12:19.323851 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323788    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="gather"
Sep 05 03:12:19.323851 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.323797    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b860ac9a-70bf-4514-bdf0-399a8c8cd3ec" containerName="copy"
Sep 05 03:12:19.327151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.327131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.329737 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.329716    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b5gnx\"/\"kube-root-ca.crt\""
Sep 05 03:12:19.330766 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.330730    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b5gnx\"/\"default-dockercfg-ngc74\""
Sep 05 03:12:19.330766 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.330735    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b5gnx\"/\"openshift-service-ca.crt\""
Sep 05 03:12:19.337632 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.335718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"]
Sep 05 03:12:19.444440 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.444402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2v5\" (UniqueName: \"kubernetes.io/projected/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-kube-api-access-kg2v5\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.444440 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.444446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-podres\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.444619 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.444472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-lib-modules\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.444619 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.444491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-sys\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.444619 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.444604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-proc\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545241 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2v5\" (UniqueName: \"kubernetes.io/projected/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-kube-api-access-kg2v5\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545342 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-podres\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545342 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-lib-modules\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545342 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-sys\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-sys\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-proc\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-proc\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-podres\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.545507 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.545416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-lib-modules\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.553167 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.553150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2v5\" (UniqueName: \"kubernetes.io/projected/c23cb96d-08b3-4edb-81e7-cd9af1a0bde0-kube-api-access-kg2v5\") pod \"perf-node-gather-daemonset-2fnj6\" (UID: \"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.642186 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.642168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:19.908104 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:19.908081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"]
Sep 05 03:12:19.910666 ip-10-0-140-13 kubenswrapper[2560]: W0905 03:12:19.910636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc23cb96d_08b3_4edb_81e7_cd9af1a0bde0.slice/crio-dd85c36a8421e11e3489df33501c9b46c643bb5038b1639d5910c3598f7a3f54 WatchSource:0}: Error finding container dd85c36a8421e11e3489df33501c9b46c643bb5038b1639d5910c3598f7a3f54: Status 404 returned error can't find the container with id dd85c36a8421e11e3489df33501c9b46c643bb5038b1639d5910c3598f7a3f54
Sep 05 03:12:20.169732 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.169663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6" event={"ID":"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0","Type":"ContainerStarted","Data":"6a468e1501037476d2efaa468a57ebc94201305330058ec1145f62a6fc9448c0"}
Sep 05 03:12:20.169732 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.169711    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6" event={"ID":"c23cb96d-08b3-4edb-81e7-cd9af1a0bde0","Type":"ContainerStarted","Data":"dd85c36a8421e11e3489df33501c9b46c643bb5038b1639d5910c3598f7a3f54"}
Sep 05 03:12:20.170117 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.169782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:20.185934 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.185890    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6" podStartSLOduration=1.185876221 podStartE2EDuration="1.185876221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:12:20.18396837 +0000 UTC m=+3283.823180171" watchObservedRunningTime="2026-09-05 03:12:20.185876221 +0000 UTC m=+3283.825088200"
Sep 05 03:12:20.489000 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.488911    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qf7hv_17eded11-bbfc-4a5e-a7d7-f22392d7a5f8/dns/0.log"
Sep 05 03:12:20.509231 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.509190    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qf7hv_17eded11-bbfc-4a5e-a7d7-f22392d7a5f8/kube-rbac-proxy/0.log"
Sep 05 03:12:20.557454 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:20.557435    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r92cr_fb62105b-3282-459d-a241-fb10b3ae0191/dns-node-resolver/0.log"
Sep 05 03:12:21.086506 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:21.086472    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7n8lq_b29d312f-67d2-4530-bcfd-b6e7f1dada9c/node-ca/0.log"
Sep 05 03:12:22.208784 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.208734    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5jd8_f0078607-a4d5-4899-825f-ea220eb8f185/serve-healthcheck-canary/0.log"
Sep 05 03:12:22.617042 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.616962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-hp7jp_21a849dc-df64-4332-803b-45cdb526a487/insights-operator/0.log"
Sep 05 03:12:22.617803 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.617771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-hp7jp_21a849dc-df64-4332-803b-45cdb526a487/insights-operator/1.log"
Sep 05 03:12:22.700543 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.700520    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mnk4_6cbf3b6f-e776-4bd5-8d0d-354cd5314481/kube-rbac-proxy/0.log"
Sep 05 03:12:22.720686 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.720663    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mnk4_6cbf3b6f-e776-4bd5-8d0d-354cd5314481/exporter/0.log"
Sep 05 03:12:22.742434 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:22.742415    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mnk4_6cbf3b6f-e776-4bd5-8d0d-354cd5314481/extractor/0.log"
Sep 05 03:12:25.677333 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.677300    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2fzbg_bf15b75c-2d31-4bdb-ae98-9968708590a2/s3-init/0.log"
Sep 05 03:12:25.692537 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.692513    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2fzbg_bf15b75c-2d31-4bdb-ae98-9968708590a2/download-hf-model/0.log"
Sep 05 03:12:25.724824 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.724793    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-mh47x_462ac25e-9aa3-416a-994d-27b724837786/s3-tls-init-custom/0.log"
Sep 05 03:12:25.773329 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.773309    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-mh47x_462ac25e-9aa3-416a-994d-27b724837786/download-hf-model/0.log"
Sep 05 03:12:25.800833 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.800810    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-jt2hp_9cf9d1cc-5907-448c-abe4-f837c7aafc16/s3-tls-init-serving/0.log"
Sep 05 03:12:25.817373 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.817356    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-jt2hp_9cf9d1cc-5907-448c-abe4-f837c7aafc16/download-hf-model/0.log"
Sep 05 03:12:25.869721 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:25.869696    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-k6ftj_95e83c31-5be6-4897-9482-f68df1952b45/seaweedfs-tls-custom/0.log"
Sep 05 03:12:26.183151 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:26.183129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-2fnj6"
Sep 05 03:12:30.457765 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:30.457710    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vltzw_77ab1399-a71b-48c6-83d3-0b1d33539384/kube-storage-version-migrator-operator/1.log"
Sep 05 03:12:30.459826 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:30.459799    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vltzw_77ab1399-a71b-48c6-83d3-0b1d33539384/kube-storage-version-migrator-operator/0.log"
Sep 05 03:12:31.526431 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.526399    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:12:31.556627 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.556589    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/egress-router-binary-copy/0.log"
Sep 05 03:12:31.576717 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.576694    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/cni-plugins/0.log"
Sep 05 03:12:31.599653 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.599634    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/bond-cni-plugin/0.log"
Sep 05 03:12:31.621548 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.621530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/routeoverride-cni/0.log"
Sep 05 03:12:31.676585 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.676562    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/whereabouts-cni-bincopy/0.log"
Sep 05 03:12:31.695866 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.695846    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2hw6_5e624dab-2c86-455c-a2f0-78752f63e187/whereabouts-cni/0.log"
Sep 05 03:12:31.946024 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:31.945999    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x2bng_315f69fb-bac2-49cc-b31a-a66da307c403/kube-multus/0.log"
Sep 05 03:12:32.053449 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:32.053424    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mj9x2_fcc83524-5eb0-4089-8aae-08ff8edd38c6/network-metrics-daemon/0.log"
Sep 05 03:12:32.069546 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:32.069525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mj9x2_fcc83524-5eb0-4089-8aae-08ff8edd38c6/kube-rbac-proxy/0.log"
Sep 05 03:12:33.062447 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.062418    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-controller/0.log"
Sep 05 03:12:33.077386 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.077361    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/0.log"
Sep 05 03:12:33.077386 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.077401    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovn-acl-logging/1.log"
Sep 05 03:12:33.097237 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.097213    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/kube-rbac-proxy-node/0.log"
Sep 05 03:12:33.116675 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.116651    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 03:12:33.133733 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.133706    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/northd/0.log"
Sep 05 03:12:33.153846 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.153818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/nbdb/0.log"
Sep 05 03:12:33.174965 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.174944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/sbdb/0.log"
Sep 05 03:12:33.241677 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:33.241652    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncjl_72c887cb-e8a9-4efc-83cd-46f1220a2c52/ovnkube-controller/0.log"
Sep 05 03:12:34.414732 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:34.414697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lb7h9_e66d0bea-4ac7-4181-a160-1f1b3b347c13/network-check-target-container/0.log"
Sep 05 03:12:35.341628 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:35.341591    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-cgmc8_cf2e9965-c0c1-4682-9359-b5a22dbc0f89/iptables-alerter/0.log"
Sep 05 03:12:36.035006 ip-10-0-140-13 kubenswrapper[2560]: I0905 03:12:36.034976    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mwksw_104bcdef-0a06-49f0-b2f8-6fadfc3950a4/tuned/0.log"