Sep 17 17:54:33.291333 ip-10-0-133-151 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:54:33.291342 ip-10-0-133-151 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:54:33.291349 ip-10-0-133-151 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:54:33.291587 ip-10-0-133-151 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:54:43.312829 ip-10-0-133-151 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:54:43.312848 ip-10-0-133-151 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 76f25a0b57de44878b3d49b2ed1cca7d --
Sep 17 17:56:26.986130 ip-10-0-133-151 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:56:30.754859 ip-10-0-133-151 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:30.754859 ip-10-0-133-151 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:56:30.754859 ip-10-0-133-151 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:30.754859 ip-10-0-133-151 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:56:30.754859 ip-10-0-133-151 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:30.755854 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.755768    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:56:30.759074 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759057    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.759074 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759072    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.759074 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759076    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759079    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759083    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759086    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759089    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759092    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759096    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759098    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759101    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759104    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759107    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759109    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759112    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759115    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759118    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759121    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759124    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759126    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759129    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759132    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.759178 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759134    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759137    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759140    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759143    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759145    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759148    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759151    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759154    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759157    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759159    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759162    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759165    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759167    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759171    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759174    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759177    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759180    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759184    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759187    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759189    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.759697 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759192    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759195    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759198    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759201    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759203    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759206    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759209    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759212    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759215    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759218    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759221    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759224    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759227    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759229    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759233    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759236    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759239    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759242    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759244    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.760288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759247    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759250    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759254    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759259    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759262    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759265    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759268    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759272    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759275    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759278    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759281    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759284    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759287    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759290    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759293    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759296    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759298    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759301    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759304    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759307    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.760776 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759309    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759312    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759315    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759318    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759321    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759323    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759326    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759330    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759334    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759337    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759340    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759343    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759346    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759349    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759353    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759357    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759360    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.759364    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760104    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.761331 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760112    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760115    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760118    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760121    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760124    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760127    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760130    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760133    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760137    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760140    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760143    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760146    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760149    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760152    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760155    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760157    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760160    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760163    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760166    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.761839 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760169    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760172    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760175    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760177    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760181    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760184    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760188    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760192    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760195    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760198    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760200    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760203    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760208    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760211    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760214    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760217    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760220    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760223    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760227    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760229    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.762359 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760232    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760235    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760238    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760241    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760244    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760247    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760250    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760253    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760256    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760259    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760261    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760264    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760267    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760270    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760273    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760275    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760278    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760281    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760283    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760286    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.762905 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760289    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760292    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760297    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760301    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760305    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760309    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760312    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760316    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760319    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760322    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760324    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760327    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760330    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760333    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760335    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760338    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760341    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760344    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760347    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.763428 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760350    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760352    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760355    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760358    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760361    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760363    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760366    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760369    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760371    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760374    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760377    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760380    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760382    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760385    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760388    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760391    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760396    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760401    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760405    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.763945 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760408    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760782    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760788    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760792    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760795    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760799    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760802    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760805    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760808    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760811    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760814    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760817    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760820    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760823    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760841    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760845    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760848    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760851    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760853    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760856    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.764441 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760859    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760862    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760866    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760870    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760873    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760876    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760879    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760883    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760886    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760888    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760892    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760895    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760898    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760902    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760905    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760908    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760911    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760915    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760917    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.765037 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760920    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760923    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760926    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760929    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760932    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760935    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760938    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760941    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760944    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760947    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760950    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760952    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760955    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760958    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760960    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760963    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760966    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760968    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760971    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760973    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.765536 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760977    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760979    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760982    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760985    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760988    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760991    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760994    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760996    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.760999    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761002    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761005    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761007    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761010    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761013    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761016    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761018    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761021    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761023    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761030    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761033    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.766073 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761037    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761040    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761043    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761045    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761048    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761051    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761054    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761057    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761059    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761062    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761065    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761068    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761071    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761073    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761077    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761082    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761085    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761088    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761091    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.766600 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761094    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761097    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761175    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761210    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761216    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761221    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761226    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761229    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761233    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761238    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761241    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761244    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761249    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761253    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761257    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761260    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761263    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761267    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761270    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761273    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761277    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761280    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761284    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761287    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:56:30.767115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761291    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761294    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761297    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761301    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761305    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761308    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761311    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761314    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761318    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761321    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761325    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761328    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761331    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761334    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761341    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761344    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761348    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761351    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761355    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761358    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761361    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761367    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761370    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761373    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761377    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:56:30.767745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761380    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761383    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761386    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761389    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761392    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761396    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761399    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761403    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761406    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761409    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761412    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761415    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761419    2577 flags.go:64] FLAG: --help="false"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761422    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-151.ec2.internal"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761425    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761429    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761432    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761435    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761439    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761442    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761445    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761449    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761452    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:56:30.768418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761455    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761459    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761462    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761465    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761468    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761473    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761476    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761479    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761482    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761485    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761489    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761494    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761498    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761501    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761504    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761507    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761511    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761514    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761517    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761521    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761524    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761529    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761532    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761535    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761538    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:56:30.769086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761541    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761544    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761547    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761550    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761557    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761561    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761564    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761567    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761570    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761576    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761579    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761583    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761586    2577 flags.go:64] FLAG: --port="10250"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761590    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761594    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f506f3569ef2ecc2"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761597    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761601    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761604    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761607    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761611    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761614    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761617    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761620    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761624    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761627    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:56:30.769757 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761631    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761634    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761637    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761640    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761643    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761647    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761650    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761653    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761656    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761659    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761662    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761665    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761669    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761672    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761675    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761678    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761681    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761684    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761690    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761693    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761697    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761702    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761706    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761709    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761712    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:56:30.770466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761715    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761718    2577 flags.go:64] FLAG: --v="2"
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761722    2577 flags.go:64] FLAG: --version="false"
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761729    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761733    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.761736    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761867    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761871    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761874    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761877    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761880    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761883    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761886    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761890    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761892    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761895    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761898    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761901    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761904    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761907    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.771162 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761911    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761915    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761918    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761921    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761924    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761927    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761930    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761935    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761938    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761940    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761944    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761947    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761950    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761953    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761956    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761960    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761964    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761967    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761971    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761974    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.771696 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761977    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761980    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761983    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761987    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761991    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761994    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.761997    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762008    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762012    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762015    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762019    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762022    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762025    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762028    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762031    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762033    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762036    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762039    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762042    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.772231 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762046    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762049    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762052    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762054    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762058    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762061    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762064    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762067    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762070    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762072    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762075    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762078    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762081    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762084    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762086    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762089    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762092    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762095    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762097    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762100    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.772769 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762103    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762106    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762109    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762113    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762117    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762119    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762122    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762125    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762128    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762130    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762134    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762136    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762141    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762160    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762164    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762168    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762172    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762175    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762178    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762181    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.773311 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762184    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762187    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762190    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762193    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762196    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.762199    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.762782    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770032    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770048    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770069    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770079    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770136    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770142    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770146    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770150    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.773837 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770153    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770156    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770160    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770163    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770166    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770169    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770172    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770175    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770178    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770181    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770184    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770187    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770189    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770192    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770196    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770199    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770202    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770204    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770209    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.774259 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770213    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770216    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770220    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770223    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770225    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770228    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770233    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770236    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770239    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770242    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770247    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770251    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770254    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770258    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770261    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770264    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770268    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770271    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770274    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.774765 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770277    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770280    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770284    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770287    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770290    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770293    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770296    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770299    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770302    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770305    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770308    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770311    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770314    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770317    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770320    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770323    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770326    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770329    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770332    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770335    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.775288 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770339    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770342    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770344    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770347    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770350    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770353    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770355    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770358    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770361    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770364    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770366    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770369    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770374    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770378    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770381    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770384    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770387    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770390    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770393    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.775882 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770396    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770398    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770401    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770404    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770407    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770410    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770413    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770416    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770418    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770421    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770424    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770427    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770430    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770433    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770437    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770440    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770443    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.776384 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770446    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770451    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770571    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770576    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770579    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770582    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770585    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770588    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770591    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770595    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770600    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770603    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770607    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770610    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:30.776927 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770613    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770616    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770619    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770622    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770625    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770628    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770631    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770633    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770636    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770639    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770642    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770645    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770648    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770651    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770654    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770657    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770660    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770663    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770673    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770677    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:30.777373 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770680    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770683    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770686    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770689    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770692    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770695    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770699    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770702    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770705    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770708    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770712    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770715    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770717    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770720    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770723    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770726    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770729    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770732    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770734    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770737    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:30.778134 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770740    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770743    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770746    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770748    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770751    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770754    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770757    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770761    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770764    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770769    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770772    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770775    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770779    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770781    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770784    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770787    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770790    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770793    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770795    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:30.778705 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770798    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770801    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770804    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770807    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770810    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770812    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770815    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770818    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770821    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770823    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770838    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770841    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770844    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770846    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770849    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770852    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770855    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770858    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770861    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770864    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:30.779269 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770866    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770869    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770872    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770875    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770879    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770882    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770884    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:30.770887    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770892    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.770998    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.774253    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.775691    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.775797    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:30.779799 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.775830    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:30.780178 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.779077    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:30.780393 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.780374    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:30.785635 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.785619    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:56:30.787394 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.787379    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 17:56:30.788291 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.788272    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:56:30.793076 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.793043    2577 fs.go:135] Filesystem UUIDs: map[3934e336-b760-41f6-b586-2eaa3188e1dd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d11c98e-3e52-437f-8c75-ebb61979e33a:/dev/nvme0n1p4]
Sep 17 17:56:30.796914 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.796881    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:56:30.801534 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.801413    2577 manager.go:216] Machine: {Timestamp:2026-09-17 17:56:30.800709325 +0000 UTC m=+2.125030741 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096502 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec271191437e7fe5d24426f45fe635ae SystemUUID:ec271191-437e-7fe5-d244-26f45fe635ae BootID:76f25a0b-57de-4487-8b3d-49b2ed1cca7d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:c7:ae:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:c7:ae:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:67:14:ac:9b:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:56:30.801652 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.801540    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:56:30.801742 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.801696    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:56:30.803482 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.803449    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:56:30.803648 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.803487    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-151.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:56:30.805031 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.805017    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:56:30.805104 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.805038    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:56:30.805104 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.805050    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:30.805104 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.805066    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:30.805252 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.805158    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:56:30.808290 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808276    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:30.808442 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808431    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:56:30.808519 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808474    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:56:30.808571 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808529    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:56:30.808571 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808551    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:56:30.808571 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808566    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:56:30.808700 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.808592    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:56:30.809092 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.809064    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:30.809201 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.809107    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:30.811534 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.811518    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:56:30.815058 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.815036    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:56:30.815125 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.815074    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:56:30.816354 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816340    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:56:30.816396 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816361    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:56:30.816396 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816367    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:56:30.816396 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816372    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:56:30.816396 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816381    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:56:30.816396 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816387    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816401    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816407    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816413    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816419    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816428    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816433    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816458    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:56:30.816536 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.816464    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:56:30.820103 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.820085    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-151.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:56:30.820183 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.820114    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:56:30.820351 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.820329    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-151.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:56:30.820889 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.820875    2577 server.go:1268] "Started kubelet"
Sep 17 17:56:30.820988 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.820960    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:56:30.821036 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.820966    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:56:30.821078 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.821065    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:56:30.821770 ip-10-0-133-151 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:56:30.822292 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.822272    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:56:30.822398 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.822275    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:56:30.825177 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.825156    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:30.827206 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.827189    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:56:30.831056 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831033    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:56:30.831154 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831061    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:30.831215 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.831147    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:30.831733 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831689    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:56:30.831733 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831712    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:56:30.831882 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831688    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:56:30.831946 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831897    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:56:30.831946 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.831906    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:56:30.832041 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832008    2577 factory.go:55] Registering systemd factory
Sep 17 17:56:30.832041 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832025    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:56:30.832280 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832262    2577 factory.go:153] Registering CRI-O factory
Sep 17 17:56:30.832280 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832278    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:56:30.832432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832332    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:56:30.832432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832360    2577 factory.go:103] Registering Raw factory
Sep 17 17:56:30.832432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832377    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:56:30.832869 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.832852    2577 manager.go:349] Starting recovery of all containers
Sep 17 17:56:30.833893 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.833857    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-151.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:56:30.833986 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.833936    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:56:30.859949 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.859930    2577 manager.go:354] Recovery completed
Sep 17 17:56:30.865543 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.865527    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:30.873487 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.873459    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:30.873582 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.873504    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:30.873582 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.873522    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:30.874147 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.874129    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:56:30.874147 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.874145    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:56:30.874242 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.874148    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:30.874242 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.874162    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:30.875415 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875398    2577 policy_none.go:49] "None policy: Start"
Sep 17 17:56:30.875415 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875415    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:56:30.875568 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875426    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:56:30.875568 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875440    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:30.875568 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875478    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:30.875568 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.875493    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:30.876513 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.876497    2577 policy_none.go:47] "Start"
Sep 17 17:56:30.879351 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.878033    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-151.ec2.internal.18d62d5fa9d5b0c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-151.ec2.internal,UID:ip-10-0-133-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-151.ec2.internal,},FirstTimestamp:2026-09-17 17:56:30.820847812 +0000 UTC m=+2.145169229,LastTimestamp:2026-09-17 17:56:30.820847812 +0000 UTC m=+2.145169229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-151.ec2.internal,}"
Sep 17 17:56:30.889850 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.889754    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-151.ec2.internal.18d62d5facf8e4b6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-151.ec2.internal,UID:ip-10-0-133-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-151.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-151.ec2.internal,},FirstTimestamp:2026-09-17 17:56:30.873486518 +0000 UTC m=+2.197807939,LastTimestamp:2026-09-17 17:56:30.873486518 +0000 UTC m=+2.197807939,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-151.ec2.internal,}"
Sep 17 17:56:30.909249 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.898113    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5kf6"
Sep 17 17:56:30.909249 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.900139    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-151.ec2.internal.18d62d5facf94ba9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-151.ec2.internal,UID:ip-10-0-133-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-151.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-151.ec2.internal,},FirstTimestamp:2026-09-17 17:56:30.873512873 +0000 UTC m=+2.197834294,LastTimestamp:2026-09-17 17:56:30.873512873 +0000 UTC m=+2.197834294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-151.ec2.internal,}"
Sep 17 17:56:30.909615 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.909599    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5kf6"
Sep 17 17:56:30.914396 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.914337    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-151.ec2.internal.18d62d5facf9855c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-151.ec2.internal,UID:ip-10-0-133-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-151.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-151.ec2.internal,},FirstTimestamp:2026-09-17 17:56:30.873527644 +0000 UTC m=+2.197849067,LastTimestamp:2026-09-17 17:56:30.873527644 +0000 UTC m=+2.197849067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-151.ec2.internal,}"
Sep 17 17:56:30.916404 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916391    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:56:30.916484 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.916432    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:56:30.916484 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916443    2577 server.go:87] "Starting device plugin registration server"
Sep 17 17:56:30.916662 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916649    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:56:30.916727 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916675    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:56:30.916811 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916793    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:56:30.916960 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916875    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:56:30.916960 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.916885    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:56:30.917399 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.917373    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:56:30.917483 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.917449    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:30.934597 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.934567    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:56:30.935868 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.935852    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:56:30.935923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.935904    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:56:30.935974 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.935929    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:56:30.936030 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:30.935976    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:56:30.938250 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:30.938225    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:31.018635 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.018535    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.019405 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.019384    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.019482 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.019418    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.019482 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.019428    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.019482 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.019450    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.031737 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.031717    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.031823 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.031738    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-151.ec2.internal\": node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.036953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.036933    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"]
Sep 17 17:56:31.037048 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.037004    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.037923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.037904    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.037992 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.037936    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.037992 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.037948    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.040070 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.040057    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.040204 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.040189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.040253 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.040221    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.043413 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043397    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.043504 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043399    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.043504 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043456    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.043504 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043468    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.043504 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043427    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.043718 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.043524    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.045709 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.045691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.045792 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.045718    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.046464 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.046448    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.046557 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.046478    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.046557 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.046492    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.064730 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.064708    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.070695 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.070679    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-151.ec2.internal\" not found" node="ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.075565 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.075550    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-151.ec2.internal\" not found" node="ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.164925 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.164897    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.233595 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.233565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.233595 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.233595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.233795 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.233615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e82e4d71f8fc63cce18d0028f27b2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-151.ec2.internal\" (UID: \"5e82e4d71f8fc63cce18d0028f27b2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.265584 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.265560    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.334354 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e82e4d71f8fc63cce18d0028f27b2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-151.ec2.internal\" (UID: \"5e82e4d71f8fc63cce18d0028f27b2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.334354 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.334354 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.334506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.334506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e82e4d71f8fc63cce18d0028f27b2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-151.ec2.internal\" (UID: \"5e82e4d71f8fc63cce18d0028f27b2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.334506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.334413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac28a8063d020db45c62eee2434bad2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal\" (UID: \"ac28a8063d020db45c62eee2434bad2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.367940 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.367917    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.374597 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.374583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.379162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.379145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:31.470052 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.470010    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.571140 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.571109    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.672930 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.672851    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.773227 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.773194    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.775541 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.775522    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:56:31.775690 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.775664    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:56:31.874220 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.874190    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.914450 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.914412    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:51:31 +0000 UTC" deadline="2028-03-25 00:05:23.863255595 +0000 UTC"
Sep 17 17:56:31.914450 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.914443    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13302h8m51.948816213s"
Sep 17 17:56:31.928950 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.928887    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:31.930057 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.930038    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:31.930151 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.930071    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:31.930151 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.930082    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:31.975210 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:31.975030    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:31.982152 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:31.982133    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:32.060550 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:32.060518    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e82e4d71f8fc63cce18d0028f27b2c2.slice/crio-47526d1f0783d46e0aeb1cc8d1e00850fb29fcbabae3f867a9aee0d7b1298a53 WatchSource:0}: Error finding container 47526d1f0783d46e0aeb1cc8d1e00850fb29fcbabae3f867a9aee0d7b1298a53: Status 404 returned error can't find the container with id 47526d1f0783d46e0aeb1cc8d1e00850fb29fcbabae3f867a9aee0d7b1298a53
Sep 17 17:56:32.061140 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:32.061125    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac28a8063d020db45c62eee2434bad2c.slice/crio-2afd59f56bff03cd5c6a6e00ded9479a43fe29355e7b51024a891798c9a61e7a WatchSource:0}: Error finding container 2afd59f56bff03cd5c6a6e00ded9479a43fe29355e7b51024a891798c9a61e7a: Status 404 returned error can't find the container with id 2afd59f56bff03cd5c6a6e00ded9479a43fe29355e7b51024a891798c9a61e7a
Sep 17 17:56:32.068143 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.068122    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:56:32.076122 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.076086    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:32.178464 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.178413    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:32.281316 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.281223    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:32.350541 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.350514    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:32.381695 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.381666    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-151.ec2.internal\" not found"
Sep 17 17:56:32.381858 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.381698    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:32.432679 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.432648    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:32.455118 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.455093    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:32.456158 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.456142    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal"
Sep 17 17:56:32.466675 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.466654    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:32.809457 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.809431    2577 apiserver.go:52] "Watching apiserver"
Sep 17 17:56:32.817265 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.817240    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:56:32.817683 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.817657    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-nxrr6","openshift-ovn-kubernetes/ovnkube-node-qmqtk","kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7","openshift-cluster-node-tuning-operator/tuned-wsqgd","openshift-dns/node-resolver-bxjw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal","openshift-multus/multus-additional-cni-plugins-s7bq8","openshift-multus/network-metrics-daemon-zwqgb","openshift-network-diagnostics/network-check-target-wnm46","kube-system/global-pull-secret-syncer-rkwq9","kube-system/konnectivity-agent-hrb8m","openshift-image-registry/node-ca-c4hcl","openshift-multus/multus-bkksm"]
Sep 17 17:56:32.820185 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.820165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:32.820282 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.820252    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:32.822576 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.822554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.826872 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.826125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.829507 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.828794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.830937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.829835    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:56:32.830937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.830053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.830937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.830126    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nd77d\""
Sep 17 17:56:32.830937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.830053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.830937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.830391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.833014 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.831906    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:32.833014 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.832549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.838373 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.838356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.838459 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.838432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:32.838700 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.838677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.839661 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.839641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:56:32.839764 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.839722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.839904 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.839881    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.840004 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.839988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:56:32.840065 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cxcg\""
Sep 17 17:56:32.840121 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.840166 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.840231 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840215    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:56:32.840283 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bsz4n\""
Sep 17 17:56:32.840332 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.840664 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840647    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.840952 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.840934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkw8p\""
Sep 17 17:56:32.841186 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.841276 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.841352 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.841539 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wlctl\""
Sep 17 17:56:32.841600 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:32.842004 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.841982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.842138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-modprobe-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-run\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-host\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-k8s-cni-cncf-io\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-conf-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-etc-kubernetes\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-systemd\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/214c7453-5da4-4c58-af2f-e64a308f7679-serviceca\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cnibin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-daemon-config\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrrm\" (UniqueName: \"kubernetes.io/projected/6f7488df-ca4b-445f-b7f8-f738c2e59fed-kube-api-access-kdrrm\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-conf\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-tuned\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-tmp\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-system-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-hostroot\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysconfig\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed57787-1808-4845-a93a-d2ddea2000cd-tmp-dir\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:32.842692 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-device-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-socket-dir-parent\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-multus\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-kubelet\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-multus-certs\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqvt\" (UniqueName: \"kubernetes.io/projected/ad089ea4-f78f-4532-9c1c-055c69086b6a-kube-api-access-7wqvt\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ed57787-1808-4845-a93a-d2ddea2000cd-hosts-file\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwlf2\" (UniqueName: \"kubernetes.io/projected/3ed57787-1808-4845-a93a-d2ddea2000cd-kube-api-access-jwlf2\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.842984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4x8m\" (UniqueName: \"kubernetes.io/projected/edac866d-e49b-4787-b73a-7b65d31c2946-kube-api-access-b4x8m\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/214c7453-5da4-4c58-af2f-e64a308f7679-host\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-netns\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-lib-modules\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-var-lib-kubelet\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cni-binary-copy\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.843368 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-bin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.844164 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-kubernetes\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.844164 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-sys\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.844164 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btjs\" (UniqueName: \"kubernetes.io/projected/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-kube-api-access-5btjs\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.844164 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.843299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktbw\" (UniqueName: \"kubernetes.io/projected/214c7453-5da4-4c58-af2f-e64a308f7679-kube-api-access-gktbw\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.844826 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.844808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:32.844910 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.844873    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:32.846437 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.846416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:56:32.851326 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.846437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:56:32.851451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.851429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.851822 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.851804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:32.851894 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.851871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:56:32.852307 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.852289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:56:32.852374 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.852347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hl8j9\""
Sep 17 17:56:32.852514 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.852492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:56:32.852645 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.852626    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:32.852777 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.852350    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.853043 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.853025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:56:32.853120 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.853066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x47xl\""
Sep 17 17:56:32.853318 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.853253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:56:32.853390 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.853339    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:56:32.853705 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.853460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg25s\""
Sep 17 17:56:32.854790 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.854475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kvrll\""
Sep 17 17:56:32.854790 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.854531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:56:32.854790 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.854482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:56:32.856295 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.856117    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:32.906811 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.906788    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:32.933480 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.933459    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:56:32.935155 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.935136    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4ndxj"
Sep 17 17:56:32.941492 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.941475    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4ndxj"
Sep 17 17:56:32.942471 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.942408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal" event={"ID":"ac28a8063d020db45c62eee2434bad2c","Type":"ContainerStarted","Data":"2afd59f56bff03cd5c6a6e00ded9479a43fe29355e7b51024a891798c9a61e7a"}
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-kubelet\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-modprobe-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-host\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-etc-kubernetes\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3045519-3c50-4afd-84ed-1d0a67387ef0-konnectivity-ca\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-ovn\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-bin\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-etc-kubernetes\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-host\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-host-slash\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65n9\" (UniqueName: \"kubernetes.io/projected/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-kube-api-access-h65n9\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrrm\" (UniqueName: \"kubernetes.io/projected/6f7488df-ca4b-445f-b7f8-f738c2e59fed-kube-api-access-kdrrm\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-modprobe-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.945107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-config\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.943986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-conf\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-tuned\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-d\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal" event={"ID":"5e82e4d71f8fc63cce18d0028f27b2c2","Type":"ContainerStarted","Data":"47526d1f0783d46e0aeb1cc8d1e00850fb29fcbabae3f867a9aee0d7b1298a53"}
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-system-cni-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-slash\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-systemd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-etc-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysctl-conf\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-env-overrides\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-dbus\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:32.946029 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944835    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-socket-dir-parent\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.944844    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-multus\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-multus-certs\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovn-node-metrics-cert\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-script-lib\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqvt\" (UniqueName: \"kubernetes.io/projected/ad089ea4-f78f-4532-9c1c-055c69086b6a-kube-api-access-7wqvt\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-multus\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-var-lib-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.944996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-multus-certs\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-log-socket\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-netd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-socket-dir-parent\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-lib-modules\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cni-binary-copy\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.946935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-sys\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.945254    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:33.445223916 +0000 UTC m=+4.769545323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-sys\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-systemd-units\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-lib-modules\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-node-log\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzlz\" (UniqueName: \"kubernetes.io/projected/f7fd071e-9de8-4a2d-b721-d8f9142e7469-kube-api-access-nbzlz\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-run\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-k8s-cni-cncf-io\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-run\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-conf-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-k8s-cni-cncf-io\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-systemd\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.947829 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/214c7453-5da4-4c58-af2f-e64a308f7679-serviceca\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cni-binary-copy\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cnibin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-daemon-config\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-systemd\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-cnibin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.945994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-tmp\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-conf-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-system-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-hostroot\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-iptables-alerter-script\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysconfig\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed57787-1808-4845-a93a-d2ddea2000cd-tmp-dir\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnbm\" (UniqueName: \"kubernetes.io/projected/42511188-d864-4c34-afc0-11ad0c80708d-kube-api-access-xvnbm\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.948953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-device-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/214c7453-5da4-4c58-af2f-e64a308f7679-serviceca\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-kubelet\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-cnibin\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-daemon-config\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-kubelet\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-hostroot\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-system-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ed57787-1808-4845-a93a-d2ddea2000cd-hosts-file\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwlf2\" (UniqueName: \"kubernetes.io/projected/3ed57787-1808-4845-a93a-d2ddea2000cd-kube-api-access-jwlf2\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4x8m\" (UniqueName: \"kubernetes.io/projected/edac866d-e49b-4787-b73a-7b65d31c2946-kube-api-access-b4x8m\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/214c7453-5da4-4c58-af2f-e64a308f7679-host\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-netns\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed57787-1808-4845-a93a-d2ddea2000cd-tmp-dir\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.949853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-sysconfig\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-kubelet-config\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-var-lib-kubelet\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-bin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-netns\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-kubernetes\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.946974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btjs\" (UniqueName: \"kubernetes.io/projected/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-kube-api-access-5btjs\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktbw\" (UniqueName: \"kubernetes.io/projected/214c7453-5da4-4c58-af2f-e64a308f7679-kube-api-access-gktbw\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad089ea4-f78f-4532-9c1c-055c69086b6a-device-dir\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3045519-3c50-4afd-84ed-1d0a67387ef0-agent-certs\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-run-netns\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ed57787-1808-4845-a93a-d2ddea2000cd-hosts-file\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-var-lib-kubelet\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-multus-cni-dir\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f7488df-ca4b-445f-b7f8-f738c2e59fed-host-var-lib-cni-bin\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-kubernetes\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.947480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/214c7453-5da4-4c58-af2f-e64a308f7679-host\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.951942 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.949985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-tmp\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.951942 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.950931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-etc-tuned\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.953877 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.952648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrrm\" (UniqueName: \"kubernetes.io/projected/6f7488df-ca4b-445f-b7f8-f738c2e59fed-kube-api-access-kdrrm\") pod \"multus-bkksm\" (UID: \"6f7488df-ca4b-445f-b7f8-f738c2e59fed\") " pod="openshift-multus/multus-bkksm"
Sep 17 17:56:32.953877 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.952818    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:32.953877 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.952835    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:32.953877 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.952848    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:32.953877 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:32.952906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:56:33.452895138 +0000 UTC m=+4.777216552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:32.955682 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.955656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4x8m\" (UniqueName: \"kubernetes.io/projected/edac866d-e49b-4787-b73a-7b65d31c2946-kube-api-access-b4x8m\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:32.957635 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.957210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqvt\" (UniqueName: \"kubernetes.io/projected/ad089ea4-f78f-4532-9c1c-055c69086b6a-kube-api-access-7wqvt\") pod \"aws-ebs-csi-driver-node-xb5v7\" (UID: \"ad089ea4-f78f-4532-9c1c-055c69086b6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:32.957635 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.957598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktbw\" (UniqueName: \"kubernetes.io/projected/214c7453-5da4-4c58-af2f-e64a308f7679-kube-api-access-gktbw\") pod \"node-ca-c4hcl\" (UID: \"214c7453-5da4-4c58-af2f-e64a308f7679\") " pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:32.958270 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.957855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btjs\" (UniqueName: \"kubernetes.io/projected/9382aeb7-efd4-4f12-a5a0-6cfb20f35f43-kube-api-access-5btjs\") pod \"tuned-wsqgd\" (UID: \"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43\") " pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:32.959271 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:32.959106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwlf2\" (UniqueName: \"kubernetes.io/projected/3ed57787-1808-4845-a93a-d2ddea2000cd-kube-api-access-jwlf2\") pod \"node-resolver-bxjw4\" (UID: \"3ed57787-1808-4845-a93a-d2ddea2000cd\") " pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:33.047367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovn-node-metrics-cert\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-script-lib\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-var-lib-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-log-socket\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-netd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-var-lib-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-netd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-systemd-units\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-node-log\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-log-socket\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzlz\" (UniqueName: \"kubernetes.io/projected/f7fd071e-9de8-4a2d-b721-d8f9142e7469-kube-api-access-nbzlz\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.047580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-systemd-units\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-node-log\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-iptables-alerter-script\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.047663    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.047723    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:33.547706711 +0000 UTC m=+4.872028127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnbm\" (UniqueName: \"kubernetes.io/projected/42511188-d864-4c34-afc0-11ad0c80708d-kube-api-access-xvnbm\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-cnibin\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-kubelet-config\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048128 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-cnibin\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.047975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-script-lib\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-kubelet-config\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-netns\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3045519-3c50-4afd-84ed-1d0a67387ef0-agent-certs\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-run-netns\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-iptables-alerter-script\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-kubelet\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3045519-3c50-4afd-84ed-1d0a67387ef0-konnectivity-ca\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-ovn\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-bin\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-host-slash\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-kubelet\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65n9\" (UniqueName: \"kubernetes.io/projected/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-kube-api-access-h65n9\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-ovn\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.048921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-config\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-host-slash\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-system-cni-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-slash\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-systemd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-etc-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-env-overrides\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-cni-bin\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-dbus\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/554048ca-bb0c-47aa-abbd-8c853cb84489-dbus\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-host-slash\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-etc-openvswitch\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7fd071e-9de8-4a2d-b721-d8f9142e7469-run-systemd\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42511188-d864-4c34-afc0-11ad0c80708d-system-cni-dir\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3045519-3c50-4afd-84ed-1d0a67387ef0-konnectivity-ca\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:33.049583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.048788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovnkube-config\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.050304 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.049105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7fd071e-9de8-4a2d-b721-d8f9142e7469-env-overrides\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.050304 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.049141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.050304 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.049249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42511188-d864-4c34-afc0-11ad0c80708d-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.050444 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.050371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fd071e-9de8-4a2d-b721-d8f9142e7469-ovn-node-metrics-cert\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.050834 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.050815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3045519-3c50-4afd-84ed-1d0a67387ef0-agent-certs\") pod \"konnectivity-agent-hrb8m\" (UID: \"c3045519-3c50-4afd-84ed-1d0a67387ef0\") " pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:33.061909 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.061862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnbm\" (UniqueName: \"kubernetes.io/projected/42511188-d864-4c34-afc0-11ad0c80708d-kube-api-access-xvnbm\") pod \"multus-additional-cni-plugins-s7bq8\" (UID: \"42511188-d864-4c34-afc0-11ad0c80708d\") " pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.062354 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.062331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzlz\" (UniqueName: \"kubernetes.io/projected/f7fd071e-9de8-4a2d-b721-d8f9142e7469-kube-api-access-nbzlz\") pod \"ovnkube-node-qmqtk\" (UID: \"f7fd071e-9de8-4a2d-b721-d8f9142e7469\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.063120 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.063102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65n9\" (UniqueName: \"kubernetes.io/projected/0fd2c703-5270-479a-9cc7-bcfe8ff8b99b-kube-api-access-h65n9\") pod \"iptables-alerter-nxrr6\" (UID: \"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b\") " pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.146394 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.146368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4hcl"
Sep 17 17:56:33.161472 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.161444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bkksm"
Sep 17 17:56:33.171420 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.171397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7"
Sep 17 17:56:33.184993 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.184973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wsqgd"
Sep 17 17:56:33.192679 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.192660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxjw4"
Sep 17 17:56:33.200922 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.200903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7bq8"
Sep 17 17:56:33.208522 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.208501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:33.217994 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.217974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxrr6"
Sep 17 17:56:33.224603 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.224586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:33.451700 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.451612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:33.451864 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.451815    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:33.451913 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.451898    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:34.451872524 +0000 UTC m=+5.776193945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:33.552535 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.552505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:33.552711 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.552559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:33.552711 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552667    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:33.552827 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552710    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:33.552827 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552728    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:33.552827 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552740    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:33.552827 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:34.552725804 +0000 UTC m=+5.877047225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:33.552827 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:33.552798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:56:34.55278141 +0000 UTC m=+5.877102830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:33.785775 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:33.785673    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214c7453_5da4_4c58_af2f_e64a308f7679.slice/crio-f2c944911b10bb81675ae0c8bc85e72dc1048b2c3ba46d75bcc7022de57f399a WatchSource:0}: Error finding container f2c944911b10bb81675ae0c8bc85e72dc1048b2c3ba46d75bcc7022de57f399a: Status 404 returned error can't find the container with id f2c944911b10bb81675ae0c8bc85e72dc1048b2c3ba46d75bcc7022de57f399a
Sep 17 17:56:33.947247 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.947199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4hcl" event={"ID":"214c7453-5da4-4c58-af2f-e64a308f7679","Type":"ContainerStarted","Data":"f2c944911b10bb81675ae0c8bc85e72dc1048b2c3ba46d75bcc7022de57f399a"}
Sep 17 17:56:33.948314 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.948287    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:32 +0000 UTC" deadline="2028-06-09 12:47:25.846252306 +0000 UTC"
Sep 17 17:56:33.948377 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:33.948315    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15138h50m51.897941659s"
Sep 17 17:56:34.130857 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.130823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fd071e_9de8_4a2d_b721_d8f9142e7469.slice/crio-b52871dfe28d0a5d3aa320fa75553df93f1f0e447c39811bf50c656ed104567e WatchSource:0}: Error finding container b52871dfe28d0a5d3aa320fa75553df93f1f0e447c39811bf50c656ed104567e: Status 404 returned error can't find the container with id b52871dfe28d0a5d3aa320fa75553df93f1f0e447c39811bf50c656ed104567e
Sep 17 17:56:34.133942 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.133912    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9382aeb7_efd4_4f12_a5a0_6cfb20f35f43.slice/crio-5a260f6e94ddbc82e76cba48a6771768957d9472cc22a75439ade970f30b9432 WatchSource:0}: Error finding container 5a260f6e94ddbc82e76cba48a6771768957d9472cc22a75439ade970f30b9432: Status 404 returned error can't find the container with id 5a260f6e94ddbc82e76cba48a6771768957d9472cc22a75439ade970f30b9432
Sep 17 17:56:34.140732 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.140624    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad089ea4_f78f_4532_9c1c_055c69086b6a.slice/crio-151aaa124f1351f757f8d938f84bb18e5d04e8da2fd94f892c04b25042c5c0b8 WatchSource:0}: Error finding container 151aaa124f1351f757f8d938f84bb18e5d04e8da2fd94f892c04b25042c5c0b8: Status 404 returned error can't find the container with id 151aaa124f1351f757f8d938f84bb18e5d04e8da2fd94f892c04b25042c5c0b8
Sep 17 17:56:34.141318 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.141294    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd2c703_5270_479a_9cc7_bcfe8ff8b99b.slice/crio-83d8d537cce375c2158c8d0fbdc56c81f361025563855ae83a093d010f69f41b WatchSource:0}: Error finding container 83d8d537cce375c2158c8d0fbdc56c81f361025563855ae83a093d010f69f41b: Status 404 returned error can't find the container with id 83d8d537cce375c2158c8d0fbdc56c81f361025563855ae83a093d010f69f41b
Sep 17 17:56:34.142206 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.141946    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed57787_1808_4845_a93a_d2ddea2000cd.slice/crio-4380cf284d2165a5a4406a826192ccb082631b554c4fcbf92eb447abec1a56e6 WatchSource:0}: Error finding container 4380cf284d2165a5a4406a826192ccb082631b554c4fcbf92eb447abec1a56e6: Status 404 returned error can't find the container with id 4380cf284d2165a5a4406a826192ccb082631b554c4fcbf92eb447abec1a56e6
Sep 17 17:56:34.142955 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:56:34.142885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42511188_d864_4c34_afc0_11ad0c80708d.slice/crio-b097335d44f0140b615daa7ddb2b32a7a976a4db5eb3467be9c6a629d38a3585 WatchSource:0}: Error finding container b097335d44f0140b615daa7ddb2b32a7a976a4db5eb3467be9c6a629d38a3585: Status 404 returned error can't find the container with id b097335d44f0140b615daa7ddb2b32a7a976a4db5eb3467be9c6a629d38a3585
Sep 17 17:56:34.459219 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.459179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:34.459339 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.459301    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:34.459381 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.459350    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:36.459338146 +0000 UTC m=+7.783659548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:34.559985 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.559955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:34.560115 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.560023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:34.560163 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560110    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:34.560193 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560168    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:36.560155542 +0000 UTC m=+7.884476956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:34.560193 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560118    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:34.560193 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560193    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:34.560292 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560202    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:34.560292 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.560229    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:56:36.560222177 +0000 UTC m=+7.884543587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:34.938876 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.938842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:34.939049 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.938978    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:34.939399 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.939376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:34.939480 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.939468    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:34.939537 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.939525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:34.939611 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:34.939591    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:34.948492 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.948403    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:32 +0000 UTC" deadline="2028-02-25 12:02:54.38384391 +0000 UTC"
Sep 17 17:56:34.948492 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.948448    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12618h6m19.435400163s"
Sep 17 17:56:34.980820 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.980191    2577 generic.go:356] "Generic (PLEG): container finished" podID="ac28a8063d020db45c62eee2434bad2c" containerID="da614e5aea14d02b211e5de9cef4d7d29a19acdaa2aea73dabe43e848dbaa94c" exitCode=0
Sep 17 17:56:34.980820 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.980279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal" event={"ID":"ac28a8063d020db45c62eee2434bad2c","Type":"ContainerDied","Data":"da614e5aea14d02b211e5de9cef4d7d29a19acdaa2aea73dabe43e848dbaa94c"}
Sep 17 17:56:34.990373 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:34.990154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxjw4" event={"ID":"3ed57787-1808-4845-a93a-d2ddea2000cd","Type":"ContainerStarted","Data":"4380cf284d2165a5a4406a826192ccb082631b554c4fcbf92eb447abec1a56e6"}
Sep 17 17:56:35.007644 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.007558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7" event={"ID":"ad089ea4-f78f-4532-9c1c-055c69086b6a","Type":"ContainerStarted","Data":"151aaa124f1351f757f8d938f84bb18e5d04e8da2fd94f892c04b25042c5c0b8"}
Sep 17 17:56:35.009391 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.009326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wsqgd" event={"ID":"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43","Type":"ContainerStarted","Data":"5a260f6e94ddbc82e76cba48a6771768957d9472cc22a75439ade970f30b9432"}
Sep 17 17:56:35.019019 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.018890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"b52871dfe28d0a5d3aa320fa75553df93f1f0e447c39811bf50c656ed104567e"}
Sep 17 17:56:35.027518 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.026909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal" event={"ID":"5e82e4d71f8fc63cce18d0028f27b2c2","Type":"ContainerStarted","Data":"e01b7b214e69c850d26a4c20c5728c249b3f041c1e74429cb379ef11275e9d80"}
Sep 17 17:56:35.040367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.040336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerStarted","Data":"b097335d44f0140b615daa7ddb2b32a7a976a4db5eb3467be9c6a629d38a3585"}
Sep 17 17:56:35.050527 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.049859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-151.ec2.internal" podStartSLOduration=3.049840294 podStartE2EDuration="3.049840294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:35.049612851 +0000 UTC m=+6.373934276" watchObservedRunningTime="2026-09-17 17:56:35.049840294 +0000 UTC m=+6.374161731"
Sep 17 17:56:35.060374 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.060337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxrr6" event={"ID":"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b","Type":"ContainerStarted","Data":"83d8d537cce375c2158c8d0fbdc56c81f361025563855ae83a093d010f69f41b"}
Sep 17 17:56:35.070203 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.070173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bkksm" event={"ID":"6f7488df-ca4b-445f-b7f8-f738c2e59fed","Type":"ContainerStarted","Data":"656c9bd9442e707df527e77b46f39a8e95e9df1e7488d02b44733a3eb0dec176"}
Sep 17 17:56:35.090011 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:35.089923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrb8m" event={"ID":"c3045519-3c50-4afd-84ed-1d0a67387ef0","Type":"ContainerStarted","Data":"faea1a3c334df3415730ab97db011a4a01630d1994f3d464f4ae1bc9902b49c3"}
Sep 17 17:56:36.099125 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.098823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal" event={"ID":"ac28a8063d020db45c62eee2434bad2c","Type":"ContainerStarted","Data":"f82283da16b4a4654db224a54d73535a36d9ba0ed295be810bae79ec8dc4ce3f"}
Sep 17 17:56:36.480516 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.480432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:36.480682 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.480611    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:36.480682 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.480669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:40.480651317 +0000 UTC m=+11.804972723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:36.581020 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.580976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:36.581217 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.581082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:36.581217 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581200    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:36.581330 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581260    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:40.581243902 +0000 UTC m=+11.905565323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:36.581672 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581654    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:36.581789 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581679    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:36.581789 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581694    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:36.581789 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.581739    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:56:40.581723887 +0000 UTC m=+11.906045294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.937900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.937906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:36.938035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.938033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.938104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:36.938432 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:36.938178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:38.936577 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:38.936531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:38.937138 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:38.936674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:38.937138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:38.936944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:38.937138 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:38.937042    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:38.937138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:38.936817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:38.937363 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:38.937198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:40.515091 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.515054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:40.515564 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.515197    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:40.515564 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.515280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.515259832 +0000 UTC m=+19.839581233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:40.616108 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.616069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.616166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616246    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616249    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616270    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616281    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.616288323 +0000 UTC m=+19.940609727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:40.616347 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.616322    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.616313448 +0000 UTC m=+19.940634849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:40.936944 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.936846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:40.937098 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.936970    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:40.937098 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.937027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:40.937098 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.937088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:40.937271 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:40.937132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:40.937271 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:40.937203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:42.940004 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:42.939973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:42.940629 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:42.940600    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:42.941993 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:42.940205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:42.941993 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:42.941441    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:42.941993 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:42.940158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:42.941993 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:42.941529    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:44.940021 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:44.939992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:44.940461 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:44.940104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:44.940461 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:44.940132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:44.940461 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:44.940120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:44.940461 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:44.940225    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:44.940461 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:44.940286    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:46.936806 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:46.936770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:46.937248 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:46.936771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:46.937248 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:46.936893    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:46.937248 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:46.936770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:46.937248 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:46.936961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:46.937248 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:46.937114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:48.578118 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.578078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:48.578545 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.578226    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.578545 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.578292    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.578277246 +0000 UTC m=+35.902598657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.679228 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.679193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:48.679422 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.679249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:48.679422 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679346    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.679422 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679401    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:48.679422 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679418    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.679400389 +0000 UTC m=+36.003721793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.679422 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679422    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:48.679619 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679436    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:48.679619 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.679483    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.679468318 +0000 UTC m=+36.003789733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:48.937174 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.937141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:48.937345 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.937147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:48.937345 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.937260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:48.937468 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.937341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:48.937468 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:48.937406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:48.937558 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:48.937490    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:50.937081 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:50.937048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:50.937494 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:50.937132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:50.937494 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:50.937168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:50.937494 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:50.937199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:50.937494 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:50.937241    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:50.937494 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:50.937349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:52.133654 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.133434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxjw4" event={"ID":"3ed57787-1808-4845-a93a-d2ddea2000cd","Type":"ContainerStarted","Data":"34f1ad6297b150fb6aa2998d46456d4bb41c676ea11d8b2775c8c6f96f5b6c17"}
Sep 17 17:56:52.135153 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.135129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7" event={"ID":"ad089ea4-f78f-4532-9c1c-055c69086b6a","Type":"ContainerStarted","Data":"b2e6b20c43e20e6e1ab7104b319cb7b12b8045f0c92b913f153d0fb86388c18c"}
Sep 17 17:56:52.136795 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.136767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wsqgd" event={"ID":"9382aeb7-efd4-4f12-a5a0-6cfb20f35f43","Type":"ContainerStarted","Data":"f8c723988e5b4450d5af56a54b00243a49ecc430b6d4a2dc9ee6d270bbce76e3"}
Sep 17 17:56:52.140428 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"a7969a01a68dc25b5a0751c93e05208ebe646e2a73566b987a275e38f3210845"}
Sep 17 17:56:52.140513 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"9098cbbba086fa29c15981f754da21df8706ecacfdceb4430e21a621cd8c70df"}
Sep 17 17:56:52.140569 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"4156ae43de02b9cdedd6516e75ed7843b97016a77719a94eab74cbc23b8119ce"}
Sep 17 17:56:52.140633 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"cc9a9de72f85a3bed030cbc16f61e435b4fc87e49c15dafa078095e06783d5d6"}
Sep 17 17:56:52.140633 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"5daa0dfa6342dbf974972d9032e5d22a99a6bbcd26e65618bd9e1c837ccaede3"}
Sep 17 17:56:52.140748 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.140674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"856f73d608c1cdde97e07c74adb6019f16e4e5f750790fb364ed4fc5e755edd2"}
Sep 17 17:56:52.142098 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.142073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4hcl" event={"ID":"214c7453-5da4-4c58-af2f-e64a308f7679","Type":"ContainerStarted","Data":"8e67377c1f6755c6645feab96d578fbb980b37dbca581dce364c2095ddd28dff"}
Sep 17 17:56:52.143820 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.143792    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="af269cc520e56013f6bfacc6c969b14a5120d681ca2770bb8b428c893e2eac6e" exitCode=0
Sep 17 17:56:52.143912 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.143893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"af269cc520e56013f6bfacc6c969b14a5120d681ca2770bb8b428c893e2eac6e"}
Sep 17 17:56:52.148144 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.148116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bkksm" event={"ID":"6f7488df-ca4b-445f-b7f8-f738c2e59fed","Type":"ContainerStarted","Data":"b857799d0a7b9584b7a50244ae8b55bdc4cb87b16b1d6f92bf14628700b633c3"}
Sep 17 17:56:52.150065 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.150040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrb8m" event={"ID":"c3045519-3c50-4afd-84ed-1d0a67387ef0","Type":"ContainerStarted","Data":"405865f898b349084f19a9bd7561254a751ff12ecbf8fe492dd0ebcc6e7078bf"}
Sep 17 17:56:52.156660 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.156612    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bxjw4" podStartSLOduration=4.05862152 podStartE2EDuration="21.156596617s" totalImagesPullingTime="17.097975097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:52.155882828 +0000 UTC m=+23.480204250" watchObservedRunningTime="2026-09-17 17:56:52.156596617 +0000 UTC m=+23.480918041"
Sep 17 17:56:52.156774 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.156704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-151.ec2.internal" podStartSLOduration=20.156695618 podStartE2EDuration="20.156695618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:36.12388225 +0000 UTC m=+7.448203677" watchObservedRunningTime="2026-09-17 17:56:52.156695618 +0000 UTC m=+23.481017042"
Sep 17 17:56:52.172580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.172533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4hcl" podStartSLOduration=11.116703286 podStartE2EDuration="21.172518836s" totalImagesPullingTime="10.05581555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:52.172349259 +0000 UTC m=+23.496670682" watchObservedRunningTime="2026-09-17 17:56:52.172518836 +0000 UTC m=+23.496840259"
Sep 17 17:56:52.198253 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.198184    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bkksm" podStartSLOduration=4.035167211 podStartE2EDuration="21.19817304s" totalImagesPullingTime="17.163005829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:52.197902076 +0000 UTC m=+23.522223498" watchObservedRunningTime="2026-09-17 17:56:52.19817304 +0000 UTC m=+23.522494463"
Sep 17 17:56:52.228182 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.228143    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wsqgd" podStartSLOduration=4.099560596 podStartE2EDuration="21.228131526s" totalImagesPullingTime="17.12857093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:52.228089288 +0000 UTC m=+23.552410711" watchObservedRunningTime="2026-09-17 17:56:52.228131526 +0000 UTC m=+23.552452949"
Sep 17 17:56:52.582882 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.582859    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:56:52.928954 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.928608    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:56:52.582877692Z","UUID":"dc8b2b81-fa21-4412-810d-81c3825d97cc","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:56:52.931380 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.931356    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:56:52.931510 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.931392    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:56:52.936955 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.936930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:52.937083 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.936930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:52.937083 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:52.937050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:52.937175 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:52.936937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:52.937175 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:52.937129    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:52.937263 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:52.937238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:53.155014 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:53.154960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7" event={"ID":"ad089ea4-f78f-4532-9c1c-055c69086b6a","Type":"ContainerStarted","Data":"d658f7d76cd96b11edf5c529e672aef5f268deae9673ae266317803465598ea1"}
Sep 17 17:56:53.157267 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:53.157235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxrr6" event={"ID":"0fd2c703-5270-479a-9cc7-bcfe8ff8b99b","Type":"ContainerStarted","Data":"f51f2436d2a230227b214a1505bbb85e2341c4108e7bac901b578d55f45db58e"}
Sep 17 17:56:53.188874 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:53.188767    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hrb8m" podStartSLOduration=5.064510536 podStartE2EDuration="22.188728199s" totalImagesPullingTime="17.124217663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:52.297777419 +0000 UTC m=+23.622098832" watchObservedRunningTime="2026-09-17 17:56:53.188728199 +0000 UTC m=+24.513049628"
Sep 17 17:56:53.189090 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:53.189028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nxrr6" podStartSLOduration=5.091073534 podStartE2EDuration="22.189016634s" totalImagesPullingTime="17.0979431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:53.188993039 +0000 UTC m=+24.513314464" watchObservedRunningTime="2026-09-17 17:56:53.189016634 +0000 UTC m=+24.513338060"
Sep 17 17:56:54.162402 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.162361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7" event={"ID":"ad089ea4-f78f-4532-9c1c-055c69086b6a","Type":"ContainerStarted","Data":"c44001721f07c1adc9ea03b81d7039dc58df16fc2574c84e24bc7966e9d2ad3c"}
Sep 17 17:56:54.166642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.166612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"f97313a384b4df3dc1915616e33b934f364644168f9ada55265459bb80852d26"}
Sep 17 17:56:54.189155 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.189092    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb5v7" podStartSLOduration=4.285434417 podStartE2EDuration="23.189056383s" totalImagesPullingTime="18.903621966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:54.188890583 +0000 UTC m=+25.513212008" watchObservedRunningTime="2026-09-17 17:56:54.189056383 +0000 UTC m=+25.513377805"
Sep 17 17:56:54.936724 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.936677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:54.936724 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.936691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:54.936724 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:54.936691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:54.937084 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:54.936824    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:54.937084 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:54.936878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:54.937084 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:54.936975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:55.738674 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:55.738634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:55.739809 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:55.739742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:56.171047 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:56.171016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:56.171658 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:56.171639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hrb8m"
Sep 17 17:56:56.940266 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:56.940239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:56.940266 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:56.940262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:56.940811 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:56.940239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:56.940811 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:56.940347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:56.940811 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:56.940444    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:56.940811 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:56.940510    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:58.179021 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.178787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" event={"ID":"f7fd071e-9de8-4a2d-b721-d8f9142e7469","Type":"ContainerStarted","Data":"5609cb1921b1ca096f0ef34bcd7158ab06e4d2cc280620e42a61bf06758cc33e"}
Sep 17 17:56:58.179370 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.179115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:58.180743 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.180719    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="c9825eca5fd7c00c21387ad5765ea64e7f049a48efe2bcd09655d6ce3a59e0e7" exitCode=0
Sep 17 17:56:58.180861 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.180786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"c9825eca5fd7c00c21387ad5765ea64e7f049a48efe2bcd09655d6ce3a59e0e7"}
Sep 17 17:56:58.194125 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.194097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:58.210261 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.210222    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk" podStartSLOduration=10.071042332 podStartE2EDuration="27.210210352s" totalImagesPullingTime="17.13916802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:58.20881327 +0000 UTC m=+29.533134693" watchObservedRunningTime="2026-09-17 17:56:58.210210352 +0000 UTC m=+29.534531800"
Sep 17 17:56:58.937250 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.937212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:58.937474 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.937214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:58.937474 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:58.937349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:56:58.937474 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:58.937362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:58.937474 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:58.937450    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:58.937693 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:58.937536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:59.183571 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.183544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:59.184033 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.183614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:59.197644 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.197573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:56:59.481235 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.481159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wnm46"]
Sep 17 17:56:59.481392 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.481319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:56:59.481458 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:59.481417    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:56:59.488980 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.488955    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkwq9"]
Sep 17 17:56:59.489152 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.489059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:56:59.489209 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:59.489149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:56:59.489513 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.489490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwqgb"]
Sep 17 17:56:59.489615 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:56:59.489592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:56:59.489674 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:56:59.489659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:57:00.187488 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:00.187454    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="722b1a2e7c65e1e5f8b5660bf2e2dc89c3fd7090cc6db194115d9994cf6efcce" exitCode=0
Sep 17 17:57:00.187995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:00.187518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"722b1a2e7c65e1e5f8b5660bf2e2dc89c3fd7090cc6db194115d9994cf6efcce"}
Sep 17 17:57:00.937093 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:00.937060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:57:00.937245 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:00.937162    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:57:00.937245 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:00.937206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:00.937319 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:00.937274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:57:01.193119 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:01.192875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerStarted","Data":"06ca42b3c99ba08c5e85dfee1dad67c52ca6f229c027209ca244214791f27dab"}
Sep 17 17:57:01.936379 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:01.936291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:01.936510 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:01.936400    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:57:02.197099 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:02.197022    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="06ca42b3c99ba08c5e85dfee1dad67c52ca6f229c027209ca244214791f27dab" exitCode=0
Sep 17 17:57:02.197099 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:02.197083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"06ca42b3c99ba08c5e85dfee1dad67c52ca6f229c027209ca244214791f27dab"}
Sep 17 17:57:02.940607 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:02.940534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:57:02.940820 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:02.940657    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwqgb" podUID="edac866d-e49b-4787-b73a-7b65d31c2946"
Sep 17 17:57:02.940820 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:02.940711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:02.940945 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:02.940820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkwq9" podUID="554048ca-bb0c-47aa-abbd-8c853cb84489"
Sep 17 17:57:03.936623 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:03.936589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:03.937026 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:03.936687    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wnm46" podUID="c3a7b40f-58f6-45a0-afc0-c79d940e6b9f"
Sep 17 17:57:04.335566 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.335534    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-151.ec2.internal" event="NodeReady"
Sep 17 17:57:04.335745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.335696    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:04.387925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.387839    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5455578cf6-b66qs"]
Sep 17 17:57:04.407116 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.407084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.415308 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.415284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ljtnz\""
Sep 17 17:57:04.415453 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.415318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:57:04.415453 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.415318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:57:04.418790 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.418773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:57:04.420216 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.420191    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5455578cf6-b66qs"]
Sep 17 17:57:04.426081 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.426047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:57:04.426497 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.426471    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xfvxq"]
Sep 17 17:57:04.450649 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.450625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.453369 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.453346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n54wg\""
Sep 17 17:57:04.453647 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.453627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:04.453739 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.453644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfvxq"]
Sep 17 17:57:04.454491 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.454341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:04.457138 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.457121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:04.491500 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491634 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491634 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491634 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491634 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491781 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491781 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.491781 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.491726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr5z\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.524642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.524614    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9f4z"]
Sep 17 17:57:04.549767 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.549719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.552094 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.552015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9f4z"]
Sep 17 17:57:04.554899 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.554880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:04.555010 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.554925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5wl9d\""
Sep 17 17:57:04.555268 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.555246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:04.592958 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.592881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:57:04.592958 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.592922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.592958 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.592948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.592973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.592994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.593028    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.593093    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.59307452 +0000 UTC m=+67.917395924 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3271f0-9520-4826-b061-ea071ecbdc7a-cert\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr5z\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps572\" (UniqueName: \"kubernetes.io/projected/5f3271f0-9520-4826-b061-ea071ecbdc7a-kube-api-access-ps572\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.593458    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.593477    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.593541    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:05.093523245 +0000 UTC m=+36.417844663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:04.593618 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.593889 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.593632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.594122 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.594100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.596802 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.596783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.596910 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.596814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.624119 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.624092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.625072 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.625049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr5z\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:04.694367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c061ac52-ce91-447d-86da-66497b456a39-config-volume\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.694367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c061ac52-ce91-447d-86da-66497b456a39-tmp-dir\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3271f0-9520-4826-b061-ea071ecbdc7a-cert\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694530    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:04.694605 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694608    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret podName:554048ca-bb0c-47aa-abbd-8c853cb84489 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.694589771 +0000 UTC m=+68.018911194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret") pod "global-pull-secret-syncer-rkwq9" (UID: "554048ca-bb0c-47aa-abbd-8c853cb84489") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz2f\" (UniqueName: \"kubernetes.io/projected/c061ac52-ce91-447d-86da-66497b456a39-kube-api-access-rxz2f\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694678    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.694682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps572\" (UniqueName: \"kubernetes.io/projected/5f3271f0-9520-4826-b061-ea071ecbdc7a-kube-api-access-ps572\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694697    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694710    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nvrwn for pod openshift-network-diagnostics/network-check-target-wnm46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.694928 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.694775    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn podName:c3a7b40f-58f6-45a0-afc0-c79d940e6b9f nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.694746098 +0000 UTC m=+68.019067499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvrwn" (UniqueName: "kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn") pod "network-check-target-wnm46" (UID: "c3a7b40f-58f6-45a0-afc0-c79d940e6b9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.696969 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.696948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3271f0-9520-4826-b061-ea071ecbdc7a-cert\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.715184 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.715153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps572\" (UniqueName: \"kubernetes.io/projected/5f3271f0-9520-4826-b061-ea071ecbdc7a-kube-api-access-ps572\") pod \"ingress-canary-xfvxq\" (UID: \"5f3271f0-9520-4826-b061-ea071ecbdc7a\") " pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.762434 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.762399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfvxq"
Sep 17 17:57:04.795786 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.795673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz2f\" (UniqueName: \"kubernetes.io/projected/c061ac52-ce91-447d-86da-66497b456a39-kube-api-access-rxz2f\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.796078 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.795858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c061ac52-ce91-447d-86da-66497b456a39-config-volume\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.796078 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.795887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.796078 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.795939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c061ac52-ce91-447d-86da-66497b456a39-tmp-dir\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.796442 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.796129    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:04.796442 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:04.796190    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:05.296171009 +0000 UTC m=+36.620492416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:04.796442 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.796204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c061ac52-ce91-447d-86da-66497b456a39-tmp-dir\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.796442 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.796397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c061ac52-ce91-447d-86da-66497b456a39-config-volume\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.810132 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.810101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz2f\" (UniqueName: \"kubernetes.io/projected/c061ac52-ce91-447d-86da-66497b456a39-kube-api-access-rxz2f\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:04.839543 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.839506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:04.936293 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.936220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:57:04.936458 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.936433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:04.939037 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.939013    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-74sx5\""
Sep 17 17:57:04.939395 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.939025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:04.939457 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:04.939419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:05.052262 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.052223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:05.097367 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.097331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:05.097565 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:05.097506    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:05.097565 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:05.097529    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:05.097685 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:05.097593    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:06.097573961 +0000 UTC m=+37.421895364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:05.104931 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.104902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfvxq"]
Sep 17 17:57:05.108083 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:57:05.108058    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3271f0_9520_4826_b061_ea071ecbdc7a.slice/crio-fe367d713da90305f1062e8af909a10b16eb3d072d665a687f5020239e02ae5f WatchSource:0}: Error finding container fe367d713da90305f1062e8af909a10b16eb3d072d665a687f5020239e02ae5f: Status 404 returned error can't find the container with id fe367d713da90305f1062e8af909a10b16eb3d072d665a687f5020239e02ae5f
Sep 17 17:57:05.205793 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.205697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfvxq" event={"ID":"5f3271f0-9520-4826-b061-ea071ecbdc7a","Type":"ContainerStarted","Data":"fe367d713da90305f1062e8af909a10b16eb3d072d665a687f5020239e02ae5f"}
Sep 17 17:57:05.299813 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.299779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:05.299970 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:05.299895    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:05.300024 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:05.299973    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:06.29995095 +0000 UTC m=+37.624272353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:05.936497 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.936465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:05.939506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.939440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2jtj\""
Sep 17 17:57:05.939506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.939485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:05.939937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:05.939440    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:06.105656 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:06.105621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:06.105866 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:06.105761    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:06.105866 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:06.105777    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:06.105866 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:06.105844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:08.105825041 +0000 UTC m=+39.430146457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:06.306717 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:06.306632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:06.306884 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:06.306784    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:06.306884 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:06.306847    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:08.306830891 +0000 UTC m=+39.631152291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:08.119485 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:08.119443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:08.119850 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:08.119565    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:08.119850 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:08.119578    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:08.119850 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:08.119630    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:12.119614282 +0000 UTC m=+43.443935684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:08.321079 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:08.321041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:08.321237 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:08.321213    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:08.321304 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:08.321293    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:12.321269448 +0000 UTC m=+43.645590874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:08.914473 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:08.914438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.028168 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.028137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.134579 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.134548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.217349 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.217270    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="b7b1e4be58b9f00e57d043aa5f095e92e7d699870a7e2f7249e0ae6f88232abd" exitCode=0
Sep 17 17:57:09.217482 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.217357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"b7b1e4be58b9f00e57d043aa5f095e92e7d699870a7e2f7249e0ae6f88232abd"}
Sep 17 17:57:09.218902 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.218883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfvxq" event={"ID":"5f3271f0-9520-4826-b061-ea071ecbdc7a","Type":"ContainerStarted","Data":"2911d6b02f61fa5f52e2f2d961783c7c5908dbe49dc86cb0ad2e84af0ecdb322"}
Sep 17 17:57:09.268273 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.268214    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xfvxq" podStartSLOduration=2.177773832 podStartE2EDuration="5.268199961s" totalImagesPullingTime="3.090426129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:09.268119184 +0000 UTC m=+40.592440608" watchObservedRunningTime="2026-09-17 17:57:09.268199961 +0000 UTC m=+40.592521384"
Sep 17 17:57:09.661766 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.661715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.784153 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.784125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.897676 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:09.897640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:10.042990 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:10.042908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:10.224572 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:10.224539    2577 generic.go:356] "Generic (PLEG): container finished" podID="42511188-d864-4c34-afc0-11ad0c80708d" containerID="f543de30e8ce6e31772fac95bdffafd05c07df573ed303b0647aed10f405800d" exitCode=0
Sep 17 17:57:10.225079 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:10.224619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerDied","Data":"f543de30e8ce6e31772fac95bdffafd05c07df573ed303b0647aed10f405800d"}
Sep 17 17:57:11.229997 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:11.229960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" event={"ID":"42511188-d864-4c34-afc0-11ad0c80708d","Type":"ContainerStarted","Data":"cd9f8804bd29182b730d486cca5e7d711e69f3c20e1ce99090bd825c2eb0d498"}
Sep 17 17:57:11.261438 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:11.261384    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s7bq8" podStartSLOduration=9.53435589 podStartE2EDuration="40.261369389s" totalImagesPullingTime="30.727013499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:11.259884579 +0000 UTC m=+42.584206001" watchObservedRunningTime="2026-09-17 17:57:11.261369389 +0000 UTC m=+42.585690811"
Sep 17 17:57:11.387511 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:11.387468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:11.491260 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:11.491194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:12.150159 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:12.150111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:12.150348 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:12.150230    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:12.150348 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:12.150243    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:12.150348 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:12.150290    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.150277662 +0000 UTC m=+51.474599062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:12.351169 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:12.351134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:12.351546 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:12.351255    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:12.351546 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:12.351316    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.351295883 +0000 UTC m=+51.675617303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:12.950236 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:12.950191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:14.169152 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:14.169107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.495393 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:16.495365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.591959 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:16.591934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:20.203965 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:20.203918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:20.204435 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:20.204072    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:20.204435 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:20.204093    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:20.204435 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:20.204149    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.204134223 +0000 UTC m=+67.528455625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:20.405232 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:20.405201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:20.405384 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:20.405306    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:20.405384 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:20.405356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.405343873 +0000 UTC m=+67.729665273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:20.475884 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:20.475788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:20.670233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:20.670190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:22.876656 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:22.876617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:23.059831 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:23.059797    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:25.452609 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:25.452573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:25.539038 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:25.539000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:31.211054 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:31.211025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmqtk"
Sep 17 17:57:31.392923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:31.392891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:31.495944 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:31.495858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:35.911400 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:35.911359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:36.005429 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.005403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:36.211925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.211852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:57:36.212039 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.212001    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:36.212039 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.212020    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:57:36.212109 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.212077    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.212061853 +0000 UTC m=+99.536383254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:57:36.413489 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.413453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:57:36.413733 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.413602    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.413733 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.413656    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.413642408 +0000 UTC m=+99.737963813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.614649 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.614615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:57:36.617411 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.617390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:36.625628 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.625607    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:57:36.625703 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:57:36.625682    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs podName:edac866d-e49b-4787-b73a-7b65d31c2946 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:40.625664248 +0000 UTC m=+131.949985668 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs") pod "network-metrics-daemon-zwqgb" (UID: "edac866d-e49b-4787-b73a-7b65d31c2946") : secret "metrics-daemon-secret" not found
Sep 17 17:57:36.715747 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.715715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:36.715890 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.715769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:36.719564 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.719544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:36.719623 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.719581    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:36.728804 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.728783    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:36.729082 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.729066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/554048ca-bb0c-47aa-abbd-8c853cb84489-original-pull-secret\") pod \"global-pull-secret-syncer-rkwq9\" (UID: \"554048ca-bb0c-47aa-abbd-8c853cb84489\") " pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:36.738533 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.738515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrwn\" (UniqueName: \"kubernetes.io/projected/c3a7b40f-58f6-45a0-afc0-c79d940e6b9f-kube-api-access-nvrwn\") pod \"network-check-target-wnm46\" (UID: \"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f\") " pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:36.754495 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.754461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkwq9"
Sep 17 17:57:36.851511 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.851486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2jtj\""
Sep 17 17:57:36.858980 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:36.858964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:37.065401 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:37.065366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkwq9"]
Sep 17 17:57:37.067896 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:57:37.067867    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554048ca_bb0c_47aa_abbd_8c853cb84489.slice/crio-6152b8101206e7bb648615d55ca8f178eabfbdc38e7c7dd6f0e4ac08c9d8438a WatchSource:0}: Error finding container 6152b8101206e7bb648615d55ca8f178eabfbdc38e7c7dd6f0e4ac08c9d8438a: Status 404 returned error can't find the container with id 6152b8101206e7bb648615d55ca8f178eabfbdc38e7c7dd6f0e4ac08c9d8438a
Sep 17 17:57:37.153188 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:37.153119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wnm46"]
Sep 17 17:57:37.157589 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:57:37.157565    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a7b40f_58f6_45a0_afc0_c79d940e6b9f.slice/crio-cf89b3a939f0a495ee311c59f0965c32f394f6fdb29d4f6bfb4dae5e6d154cfe WatchSource:0}: Error finding container cf89b3a939f0a495ee311c59f0965c32f394f6fdb29d4f6bfb4dae5e6d154cfe: Status 404 returned error can't find the container with id cf89b3a939f0a495ee311c59f0965c32f394f6fdb29d4f6bfb4dae5e6d154cfe
Sep 17 17:57:37.289784 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:37.289739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkwq9" event={"ID":"554048ca-bb0c-47aa-abbd-8c853cb84489","Type":"ContainerStarted","Data":"6152b8101206e7bb648615d55ca8f178eabfbdc38e7c7dd6f0e4ac08c9d8438a"}
Sep 17 17:57:37.290852 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:37.290831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wnm46" event={"ID":"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f","Type":"ContainerStarted","Data":"cf89b3a939f0a495ee311c59f0965c32f394f6fdb29d4f6bfb4dae5e6d154cfe"}
Sep 17 17:57:42.304280 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:42.304115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wnm46" event={"ID":"c3a7b40f-58f6-45a0-afc0-c79d940e6b9f","Type":"ContainerStarted","Data":"a49787605825c4a8bd0f44b450cdc3af22bd96076aabe72019aa3c056052a5c8"}
Sep 17 17:57:42.304280 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:42.304211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:57:42.305733 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:42.305712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkwq9" event={"ID":"554048ca-bb0c-47aa-abbd-8c853cb84489","Type":"ContainerStarted","Data":"b62c0332d54c4458846bc3cb50d83a6a5ecb03ae8524738f60c3b41b9c8f5bc8"}
Sep 17 17:57:42.357557 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:42.357515    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rkwq9" podStartSLOduration=66.520045016 podStartE2EDuration="1m11.357504017s" totalImagesPullingTime="4.837459001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:42.357294365 +0000 UTC m=+73.681615787" watchObservedRunningTime="2026-09-17 17:57:42.357504017 +0000 UTC m=+73.681825437"
Sep 17 17:57:42.357795 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:42.357773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wnm46" podStartSLOduration=66.612004852 podStartE2EDuration="1m11.357748597s" totalImagesPullingTime="4.745743745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:42.332645993 +0000 UTC m=+73.656967417" watchObservedRunningTime="2026-09-17 17:57:42.357748597 +0000 UTC m=+73.682070025"
Sep 17 17:57:45.861479 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:45.861445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:45.945604 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:45.945571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:47.150343 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:47.150310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:47.361301 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:57:47.361272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:03.385351 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.385314    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"]
Sep 17 17:58:03.388202 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.388169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.399135 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.399113    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:58:03.399239 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.399147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:58:03.399539 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.399524    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:03.400051 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.400038    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hknxz\""
Sep 17 17:58:03.400149 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.400134    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:03.418280 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.418257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"]
Sep 17 17:58:03.492925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.492888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b43082c0-82d8-4e90-873e-17041bfaf5cd-serving-cert\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.492925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.492930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b43082c0-82d8-4e90-873e-17041bfaf5cd-config\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.493153 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.492995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbk8\" (UniqueName: \"kubernetes.io/projected/b43082c0-82d8-4e90-873e-17041bfaf5cd-kube-api-access-sqbk8\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.594331 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.594277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b43082c0-82d8-4e90-873e-17041bfaf5cd-serving-cert\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.594331 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.594333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b43082c0-82d8-4e90-873e-17041bfaf5cd-config\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.594607 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.594365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbk8\" (UniqueName: \"kubernetes.io/projected/b43082c0-82d8-4e90-873e-17041bfaf5cd-kube-api-access-sqbk8\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.595001 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.594981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b43082c0-82d8-4e90-873e-17041bfaf5cd-config\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.596553 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.596534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b43082c0-82d8-4e90-873e-17041bfaf5cd-serving-cert\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.621506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.621472    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"]
Sep 17 17:58:03.623331 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.623315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"
Sep 17 17:58:03.627211 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.627191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9d98c\""
Sep 17 17:58:03.641558 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.641503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbk8\" (UniqueName: \"kubernetes.io/projected/b43082c0-82d8-4e90-873e-17041bfaf5cd-kube-api-access-sqbk8\") pod \"service-ca-operator-57cdc4b768-h9nfc\" (UID: \"b43082c0-82d8-4e90-873e-17041bfaf5cd\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.661226 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.661197    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"]
Sep 17 17:58:03.694950 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.694914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmdr\" (UniqueName: \"kubernetes.io/projected/adf49094-f22f-46d6-9fa9-6b9589be542f-kube-api-access-6lmdr\") pod \"network-check-source-7cbc45fddc-txn98\" (UID: \"adf49094-f22f-46d6-9fa9-6b9589be542f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"
Sep 17 17:58:03.696862 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.696840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"
Sep 17 17:58:03.795816 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.795773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmdr\" (UniqueName: \"kubernetes.io/projected/adf49094-f22f-46d6-9fa9-6b9589be542f-kube-api-access-6lmdr\") pod \"network-check-source-7cbc45fddc-txn98\" (UID: \"adf49094-f22f-46d6-9fa9-6b9589be542f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"
Sep 17 17:58:03.829810 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.829773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmdr\" (UniqueName: \"kubernetes.io/projected/adf49094-f22f-46d6-9fa9-6b9589be542f-kube-api-access-6lmdr\") pod \"network-check-source-7cbc45fddc-txn98\" (UID: \"adf49094-f22f-46d6-9fa9-6b9589be542f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"
Sep 17 17:58:03.931938 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.931846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"
Sep 17 17:58:03.975664 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.975632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:03.988353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:03.988328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc"]
Sep 17 17:58:03.992815 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:03.992792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43082c0_82d8_4e90_873e_17041bfaf5cd.slice/crio-377f7ab0fb16378021363d890e07f55ac62fbdbcd4c2062b622f3d2750c8f407 WatchSource:0}: Error finding container 377f7ab0fb16378021363d890e07f55ac62fbdbcd4c2062b622f3d2750c8f407: Status 404 returned error can't find the container with id 377f7ab0fb16378021363d890e07f55ac62fbdbcd4c2062b622f3d2750c8f407
Sep 17 17:58:04.097361 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:04.097313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:04.202423 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:04.202332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98"]
Sep 17 17:58:04.206312 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:04.206289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf49094_f22f_46d6_9fa9_6b9589be542f.slice/crio-58cfe945ea7cc2cf2d8f5cf7faf45543a077d5e4a207fc7287f66ade0074a1fe WatchSource:0}: Error finding container 58cfe945ea7cc2cf2d8f5cf7faf45543a077d5e4a207fc7287f66ade0074a1fe: Status 404 returned error can't find the container with id 58cfe945ea7cc2cf2d8f5cf7faf45543a077d5e4a207fc7287f66ade0074a1fe
Sep 17 17:58:04.361867 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:04.361824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc" event={"ID":"b43082c0-82d8-4e90-873e-17041bfaf5cd","Type":"ContainerStarted","Data":"377f7ab0fb16378021363d890e07f55ac62fbdbcd4c2062b622f3d2750c8f407"}
Sep 17 17:58:04.363329 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:04.363304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98" event={"ID":"adf49094-f22f-46d6-9fa9-6b9589be542f","Type":"ContainerStarted","Data":"e0c9e9a643cda4b49db8f3d2e67c68bf167ec807770e39dec7f7862144bb1b24"}
Sep 17 17:58:04.363453 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:04.363349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98" event={"ID":"adf49094-f22f-46d6-9fa9-6b9589be542f","Type":"ContainerStarted","Data":"58cfe945ea7cc2cf2d8f5cf7faf45543a077d5e4a207fc7287f66ade0074a1fe"}
Sep 17 17:58:05.173376 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:05.173308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:05.301414 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:05.301380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:06.370253 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:06.370216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc" event={"ID":"b43082c0-82d8-4e90-873e-17041bfaf5cd","Type":"ContainerStarted","Data":"c70263fa19a094e438fe511c0755de05142ee5228b1c827059cc9df29a96a940"}
Sep 17 17:58:06.395584 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:06.395539    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc" podStartSLOduration=1.699253719 podStartE2EDuration="3.395526019s" totalImagesPullingTime="1.6962723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.395389411 +0000 UTC m=+97.719710833" watchObservedRunningTime="2026-09-17 17:58:06.395526019 +0000 UTC m=+97.719847441"
Sep 17 17:58:06.395705 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:06.395613    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-txn98" podStartSLOduration=3.39560738 podStartE2EDuration="3.39560738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:04.38598579 +0000 UTC m=+95.710307214" watchObservedRunningTime="2026-09-17 17:58:06.39560738 +0000 UTC m=+97.719928806"
Sep 17 17:58:06.852107 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:06.852038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.001978 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.001948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.153365 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.153336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.264311 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.264280    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hwhms"]
Sep 17 17:58:07.266422 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.266407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.273371 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.273353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:07.273957 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.273940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:58:07.274889 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.274874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:07.276808 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.276795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5pr9\""
Sep 17 17:58:07.277575 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.277563    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:58:07.291824 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.291803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwhms"]
Sep 17 17:58:07.320519 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.320498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.322557 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.322539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.322617 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.322567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.322617 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.322587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae9ec7ba-9975-4973-b711-50c7195153d5-crio-socket\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.322617 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.322604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ec7ba-9975-4973-b711-50c7195153d5-data-volume\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.322724 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.322685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhjv\" (UniqueName: \"kubernetes.io/projected/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-api-access-pjhjv\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423516 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhjv\" (UniqueName: \"kubernetes.io/projected/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-api-access-pjhjv\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.423640    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.423679    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae9ec7ba-9975-4973-b711-50c7195153d5-crio-socket\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.423712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.923693129 +0000 UTC m=+99.248014549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.423742    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.923729146 +0000 UTC m=+99.248050548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ec7ba-9975-4973-b711-50c7195153d5-data-volume\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.423921 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.423799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae9ec7ba-9975-4973-b711-50c7195153d5-crio-socket\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.424235 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.424042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ec7ba-9975-4973-b711-50c7195153d5-data-volume\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.456088 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.456062    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"]
Sep 17 17:58:07.457975 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.457956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"
Sep 17 17:58:07.459486 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.459467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhjv\" (UniqueName: \"kubernetes.io/projected/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-api-access-pjhjv\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.462488 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.462470    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:58:07.463143 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.463127    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:07.468596 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.468578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bzr4l\""
Sep 17 17:58:07.470008 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.469990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.474223 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.474201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"]
Sep 17 17:58:07.524194 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.524171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvmwg\" (UniqueName: \"kubernetes.io/projected/8188259b-edce-4811-84c9-66b13ee1293b-kube-api-access-bvmwg\") pod \"migrator-76794cbd6c-hnjtk\" (UID: \"8188259b-edce-4811-84c9-66b13ee1293b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"
Sep 17 17:58:07.624741 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.624710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvmwg\" (UniqueName: \"kubernetes.io/projected/8188259b-edce-4811-84c9-66b13ee1293b-kube-api-access-bvmwg\") pod \"migrator-76794cbd6c-hnjtk\" (UID: \"8188259b-edce-4811-84c9-66b13ee1293b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"
Sep 17 17:58:07.646859 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.646835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvmwg\" (UniqueName: \"kubernetes.io/projected/8188259b-edce-4811-84c9-66b13ee1293b-kube-api-access-bvmwg\") pod \"migrator-76794cbd6c-hnjtk\" (UID: \"8188259b-edce-4811-84c9-66b13ee1293b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"
Sep 17 17:58:07.660086 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.660061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.771037 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.770978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"
Sep 17 17:58:07.792290 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.792254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.927925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.927890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.927925 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:07.927924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:07.928120 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.928011    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.928120 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.928033    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.928120 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.928074    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.928059877 +0000 UTC m=+100.252381307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.928120 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:07.928088    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.92808238 +0000 UTC m=+100.252403780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.051154 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.051078    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk"]
Sep 17 17:58:08.054629 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:08.054607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8188259b_edce_4811_84c9_66b13ee1293b.slice/crio-9897baa7ec8ec1bb047574d4be53e09e811fd98aee2b12ee994f29c46b17c3f2 WatchSource:0}: Error finding container 9897baa7ec8ec1bb047574d4be53e09e811fd98aee2b12ee994f29c46b17c3f2: Status 404 returned error can't find the container with id 9897baa7ec8ec1bb047574d4be53e09e811fd98aee2b12ee994f29c46b17c3f2
Sep 17 17:58:08.229825 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.229793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") pod \"image-registry-5455578cf6-b66qs\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") " pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:58:08.229965 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.229945    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:08.229965 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.229962    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5455578cf6-b66qs: secret "image-registry-tls" not found
Sep 17 17:58:08.230027 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.230018    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls podName:1f878402-cbf9-44e8-82d2-3b47b25244c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:12.230003028 +0000 UTC m=+163.554324428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls") pod "image-registry-5455578cf6-b66qs" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5") : secret "image-registry-tls" not found
Sep 17 17:58:08.377110 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.377075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk" event={"ID":"8188259b-edce-4811-84c9-66b13ee1293b","Type":"ContainerStarted","Data":"9897baa7ec8ec1bb047574d4be53e09e811fd98aee2b12ee994f29c46b17c3f2"}
Sep 17 17:58:08.431617 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.431589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:58:08.431953 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.431713    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:58:08.431953 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.431776    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls podName:c061ac52-ce91-447d-86da-66497b456a39 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:12.431763569 +0000 UTC m=+163.756084983 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls") pod "dns-default-r9f4z" (UID: "c061ac52-ce91-447d-86da-66497b456a39") : secret "dns-default-metrics-tls" not found
Sep 17 17:58:08.935496 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.935457    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.935780 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.935357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:08.935857 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:08.935821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:08.935975 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.935955    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:10.935913378 +0000 UTC m=+102.260234797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.936050 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.936036    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.936109 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:08.936099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:10.936075185 +0000 UTC m=+102.260396600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:09.194932 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.194869    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxjw4_3ed57787-1808-4845-a93a-d2ddea2000cd/dns-node-resolver/0.log"
Sep 17 17:58:09.372980 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.372941    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:09.382698 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.382672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk" event={"ID":"8188259b-edce-4811-84c9-66b13ee1293b","Type":"ContainerStarted","Data":"ca3e28027443a8397e2606ea21fc7e1968e3c0b94dd900c91d18551fd2bc0292"}
Sep 17 17:58:09.382826 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.382717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk" event={"ID":"8188259b-edce-4811-84c9-66b13ee1293b","Type":"ContainerStarted","Data":"8effc51a6f9de063d35bda285f50af296933041fe2d686053d9f740c59ff3764"}
Sep 17 17:58:09.403796 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.403719    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-hnjtk" podStartSLOduration=1.495705997 podStartE2EDuration="2.403703613s" totalImagesPullingTime="907.997616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:09.402280496 +0000 UTC m=+100.726601918" watchObservedRunningTime="2026-09-17 17:58:09.403703613 +0000 UTC m=+100.728025036"
Sep 17 17:58:09.521415 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.521350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:09.651059 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:09.651029    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:10.190908 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:10.190879    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4hcl_214c7453-5da4-4c58-af2f-e64a308f7679/node-ca/0.log"
Sep 17 17:58:10.461117 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:10.461037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:10.952101 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:10.952072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:10.952101 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:10.952103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:10.952501 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:10.952181    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:10.952501 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:10.952221    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:10.952501 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:10.952233    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:14.952220679 +0000 UTC m=+106.276542085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:10.952501 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:10.952257    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:14.952246332 +0000 UTC m=+106.276567734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:11.207783 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:11.207697    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xfvxq_5f3271f0-9520-4826-b061-ea071ecbdc7a/serve-healthcheck-canary/0.log"
Sep 17 17:58:12.407299 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:12.407263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.543730 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:12.543700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.683156 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:12.683084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:13.269413 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:13.269384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:13.310875 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:13.310851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wnm46"
Sep 17 17:58:13.871872 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:13.871839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.446766 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:14.446721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.663884 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:14.663852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.840660 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:14.840582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.977556 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:14.977526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:14.977556 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:14.977559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:14.978030 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:14.977658    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:14.978030 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:14.977714    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:22.977701463 +0000 UTC m=+114.302022864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:14.978030 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:14.977657    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:14.978030 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:14.977815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm podName:ae9ec7ba-9975-4973-b711-50c7195153d5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:22.977797258 +0000 UTC m=+114.302118676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwhms" (UID: "ae9ec7ba-9975-4973-b711-50c7195153d5") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:15.286857 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:15.286827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.047540 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.047506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.130966 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.130939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.237893 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.237867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.464604 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.464576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.854028 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.853950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.949261 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:16.949233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.646312 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:17.646279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.865185 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:17.865154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.990427 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:17.990356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:18.095701 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:18.095672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:18.436234 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:18.436206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:18.534489 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:18.534457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.050583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:19.050545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.654009 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:19.653977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.848895 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:19.848863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.245244 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:20.245213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.334502 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:20.334470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.483802 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:20.483769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.836154 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:20.836123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:21.434375 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:21.434345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:21.834877 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:21.834800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.125584 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:22.125551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.233104 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:22.233075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.558180 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:22.558103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.690189 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:22.690163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.848663 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:22.848597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:23.042059 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.042002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:23.042059 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.042054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:23.042617 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.042577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae9ec7ba-9975-4973-b711-50c7195153d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:23.044375 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.044355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae9ec7ba-9975-4973-b711-50c7195153d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwhms\" (UID: \"ae9ec7ba-9975-4973-b711-50c7195153d5\") " pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:23.053986 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.053955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:23.174395 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.174365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwhms"
Sep 17 17:58:23.461813 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.461791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwhms"]
Sep 17 17:58:23.463429 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:23.463405    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9ec7ba_9975_4973_b711_50c7195153d5.slice/crio-d5a6efac3c3aafc7c68d7abebffe806f8311f2e1aa61441cbba1f01ec0671a7b WatchSource:0}: Error finding container d5a6efac3c3aafc7c68d7abebffe806f8311f2e1aa61441cbba1f01ec0671a7b: Status 404 returned error can't find the container with id d5a6efac3c3aafc7c68d7abebffe806f8311f2e1aa61441cbba1f01ec0671a7b
Sep 17 17:58:23.471108 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:23.471085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:24.432642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:24.432565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwhms" event={"ID":"ae9ec7ba-9975-4973-b711-50c7195153d5","Type":"ContainerStarted","Data":"5cb64270389931922695a6b0ce30f5ec8d9d0e3a0ddd91a11323c8c962a90547"}
Sep 17 17:58:24.432642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:24.432616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwhms" event={"ID":"ae9ec7ba-9975-4973-b711-50c7195153d5","Type":"ContainerStarted","Data":"de047ea43c866b33d30fe62c5a6234de608afbfa44c6e5fee546a6ceae876f4c"}
Sep 17 17:58:24.432642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:24.432634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwhms" event={"ID":"ae9ec7ba-9975-4973-b711-50c7195153d5","Type":"ContainerStarted","Data":"d5a6efac3c3aafc7c68d7abebffe806f8311f2e1aa61441cbba1f01ec0671a7b"}
Sep 17 17:58:26.442231 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:26.442197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwhms" event={"ID":"ae9ec7ba-9975-4973-b711-50c7195153d5","Type":"ContainerStarted","Data":"e1a6b3323a126216d8c1ceddf00bcb2058c18bafa626f606f449c2140a789def"}
Sep 17 17:58:26.473649 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:26.473604    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hwhms" podStartSLOduration=17.632361915 podStartE2EDuration="19.473588545s" totalImagesPullingTime="1.84122663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:26.473171522 +0000 UTC m=+117.797492945" watchObservedRunningTime="2026-09-17 17:58:26.473588545 +0000 UTC m=+117.797909965"
Sep 17 17:58:29.510419 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:29.510383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:31.568964 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:31.568934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:31.756628 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:31.756595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:37.198481 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.198448    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"]
Sep 17 17:58:37.201222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.201204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.205040 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.205019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:58:37.205040 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.205028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:58:37.205227 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.205038    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:58:37.205227 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.205028    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:58:37.205639 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.205611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rxhgr\""
Sep 17 17:58:37.218699 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.218677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"]
Sep 17 17:58:37.275872 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.275841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"]
Sep 17 17:58:37.277691 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.277673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.280598 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.280577    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"]
Sep 17 17:58:37.281142 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.281126    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:58:37.281226 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.281196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-spmvc\""
Sep 17 17:58:37.281587 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.281572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:58:37.282614 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.282590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:37.285993 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.285978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:58:37.295250 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.295232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"]
Sep 17 17:58:37.296169 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.296155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hzclj\""
Sep 17 17:58:37.304129 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.304108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"]
Sep 17 17:58:37.341888 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.341857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5c46e482-11b3-4e33-b53d-4894f0d70a44-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.342094 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.341930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c46e482-11b3-4e33-b53d-4894f0d70a44-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.342094 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.341961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.342094 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.341986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9pd\" (UniqueName: \"kubernetes.io/projected/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-kube-api-access-6d9pd\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.384283 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.384259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:37.442464 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.442430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.442464 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.442467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9pd\" (UniqueName: \"kubernetes.io/projected/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-kube-api-access-6d9pd\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.442687 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.442488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ac1636a-899a-43e8-a78e-1d6e3263fbfb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2mp6v\" (UID: \"0ac1636a-899a-43e8-a78e-1d6e3263fbfb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:37.442687 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.442535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c46e482-11b3-4e33-b53d-4894f0d70a44-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.442687 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.442559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5c46e482-11b3-4e33-b53d-4894f0d70a44-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.443286 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.443265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5c46e482-11b3-4e33-b53d-4894f0d70a44-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.444914 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.444891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5c46e482-11b3-4e33-b53d-4894f0d70a44-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ngjg7\" (UID: \"5c46e482-11b3-4e33-b53d-4894f0d70a44\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.445001 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.444979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.456184 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.456134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9pd\" (UniqueName: \"kubernetes.io/projected/b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af-kube-api-access-6d9pd\") pod \"managed-serviceaccount-addon-agent-6886d465b4-nr99z\" (UID: \"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.473374 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.473353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:37.522746 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.522723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"
Sep 17 17:58:37.542939 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.542913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ac1636a-899a-43e8-a78e-1d6e3263fbfb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2mp6v\" (UID: \"0ac1636a-899a-43e8-a78e-1d6e3263fbfb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:37.545472 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.545452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ac1636a-899a-43e8-a78e-1d6e3263fbfb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2mp6v\" (UID: \"0ac1636a-899a-43e8-a78e-1d6e3263fbfb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:37.588142 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.588116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"
Sep 17 17:58:37.593634 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.593494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:37.799453 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.799429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z"]
Sep 17 17:58:37.801724 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:37.801697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3c0a4a1_c9b2_4efb_80ec_78c49b5d16af.slice/crio-623eccd48246787856bb2ca54c2fb341277224c5cf10a8e14cd30ffecfd73f56 WatchSource:0}: Error finding container 623eccd48246787856bb2ca54c2fb341277224c5cf10a8e14cd30ffecfd73f56: Status 404 returned error can't find the container with id 623eccd48246787856bb2ca54c2fb341277224c5cf10a8e14cd30ffecfd73f56
Sep 17 17:58:37.872721 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.872694    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7"]
Sep 17 17:58:37.875665 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:37.875629    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c46e482_11b3_4e33_b53d_4894f0d70a44.slice/crio-c993cabff5976e467d02e3378e60c80a2baf942df9368d202f403927ff0b54bf WatchSource:0}: Error finding container c993cabff5976e467d02e3378e60c80a2baf942df9368d202f403927ff0b54bf: Status 404 returned error can't find the container with id c993cabff5976e467d02e3378e60c80a2baf942df9368d202f403927ff0b54bf
Sep 17 17:58:37.877878 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:37.877855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"]
Sep 17 17:58:37.880023 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:37.880000    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac1636a_899a_43e8_a78e_1d6e3263fbfb.slice/crio-2dec048defe874d8f34789377ec3e4efac42e3bfb887d2db51c010a58ece65d1 WatchSource:0}: Error finding container 2dec048defe874d8f34789377ec3e4efac42e3bfb887d2db51c010a58ece65d1: Status 404 returned error can't find the container with id 2dec048defe874d8f34789377ec3e4efac42e3bfb887d2db51c010a58ece65d1
Sep 17 17:58:38.479000 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:38.478961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v" event={"ID":"0ac1636a-899a-43e8-a78e-1d6e3263fbfb","Type":"ContainerStarted","Data":"2dec048defe874d8f34789377ec3e4efac42e3bfb887d2db51c010a58ece65d1"}
Sep 17 17:58:38.480407 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:38.480375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7" event={"ID":"5c46e482-11b3-4e33-b53d-4894f0d70a44","Type":"ContainerStarted","Data":"c993cabff5976e467d02e3378e60c80a2baf942df9368d202f403927ff0b54bf"}
Sep 17 17:58:38.481736 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:38.481712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z" event={"ID":"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af","Type":"ContainerStarted","Data":"623eccd48246787856bb2ca54c2fb341277224c5cf10a8e14cd30ffecfd73f56"}
Sep 17 17:58:39.486990 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.486620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v" event={"ID":"0ac1636a-899a-43e8-a78e-1d6e3263fbfb","Type":"ContainerStarted","Data":"b15b35170dd74a35e4909dce99152e75119e386cfcc270654dcf051cd0dae1c3"}
Sep 17 17:58:39.486990 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.486722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:39.488471 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.488438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7" event={"ID":"5c46e482-11b3-4e33-b53d-4894f0d70a44","Type":"ContainerStarted","Data":"493611b7812af1ca3fdc6d88936d3f42c34807a0a9d125c94767789ae2b6677b"}
Sep 17 17:58:39.492365 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.492333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v"
Sep 17 17:58:39.513736 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.513685    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2mp6v" podStartSLOduration=1.343910106 podStartE2EDuration="2.513666769s" totalImagesPullingTime="1.169756663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.513205748 +0000 UTC m=+130.837527172" watchObservedRunningTime="2026-09-17 17:58:39.513666769 +0000 UTC m=+130.837988187"
Sep 17 17:58:39.534357 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.534312    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ngjg7" podStartSLOduration=1.363226528 podStartE2EDuration="2.534294682s" totalImagesPullingTime="1.171068154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.533317594 +0000 UTC m=+130.857639018" watchObservedRunningTime="2026-09-17 17:58:39.534294682 +0000 UTC m=+130.858616106"
Sep 17 17:58:39.749112 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.749023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:39.903993 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:39.903965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:40.635526 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.635496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:40.674279 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.674251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:58:40.677015 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.676986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edac866d-e49b-4787-b73a-7b65d31c2946-metrics-certs\") pod \"network-metrics-daemon-zwqgb\" (UID: \"edac866d-e49b-4787-b73a-7b65d31c2946\") " pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:58:40.761338 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.761307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:40.892773 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.892681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:40.950768 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.950723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-74sx5\""
Sep 17 17:58:40.958464 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:40.958448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwqgb"
Sep 17 17:58:41.237566 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:41.237487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwqgb"]
Sep 17 17:58:41.241344 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:41.241316    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedac866d_e49b_4787_b73a_7b65d31c2946.slice/crio-23d21ca4e89301a22144c24c2de51c32704f84c0c85b18aa78169e576154292c WatchSource:0}: Error finding container 23d21ca4e89301a22144c24c2de51c32704f84c0c85b18aa78169e576154292c: Status 404 returned error can't find the container with id 23d21ca4e89301a22144c24c2de51c32704f84c0c85b18aa78169e576154292c
Sep 17 17:58:41.496698 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:41.496601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwqgb" event={"ID":"edac866d-e49b-4787-b73a-7b65d31c2946","Type":"ContainerStarted","Data":"23d21ca4e89301a22144c24c2de51c32704f84c0c85b18aa78169e576154292c"}
Sep 17 17:58:41.497968 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:41.497945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z" event={"ID":"b3c0a4a1-c9b2-4efb-80ec-78c49b5d16af","Type":"ContainerStarted","Data":"79ddacbfc7816849629bd1fe70b34dafb4794e1af38eb11fef66037e264496b4"}
Sep 17 17:58:42.502856 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.502817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwqgb" event={"ID":"edac866d-e49b-4787-b73a-7b65d31c2946","Type":"ContainerStarted","Data":"1ba7524fc949105ef5c5791d4792c0ceeef2d5129a4960164477b0f2f47239f1"}
Sep 17 17:58:42.503219 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.502894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwqgb" event={"ID":"edac866d-e49b-4787-b73a-7b65d31c2946","Type":"ContainerStarted","Data":"8634b45565cbb688eabe2a66f8b9e24e90f19ba4397e17b236da8ac762598b5f"}
Sep 17 17:58:42.526061 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.525958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6886d465b4-nr99z" podStartSLOduration=2.539650822 podStartE2EDuration="5.525940469s" totalImagesPullingTime="2.986289647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:41.525635946 +0000 UTC m=+132.849957369" watchObservedRunningTime="2026-09-17 17:58:42.525940469 +0000 UTC m=+133.850261893"
Sep 17 17:58:42.526493 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.526466    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zwqgb" podStartSLOduration=130.493381308 podStartE2EDuration="2m11.526457599s" totalImagesPullingTime="1.033076291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:42.526085389 +0000 UTC m=+133.850406795" watchObservedRunningTime="2026-09-17 17:58:42.526457599 +0000 UTC m=+133.850779021"
Sep 17 17:58:42.682948 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.682918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:42.807254 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:42.807179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:43.294461 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:43.294432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:45.359719 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:45.359689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:45.508528 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:45.508499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.097105 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.097073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.352466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.352378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.562898 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.562864    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"]
Sep 17 17:58:46.565235 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.565220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.569845 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.569824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:58:46.570694 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.570676    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:58:46.570970 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.570951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:58:46.570970 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.570966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:46.571114 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.570969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:46.571114 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.571009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mh2l6\""
Sep 17 17:58:46.589222 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.589200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"]
Sep 17 17:58:46.599574 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.599551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.635499 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.635476    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8jbj9"]
Sep 17 17:58:46.637511 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.637494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.643711 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.643694    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:46.643849 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.643784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kp7tj\""
Sep 17 17:58:46.644052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.644036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:46.644131 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.644086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:46.718902 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.718875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7799ffba-3b3f-456a-ac76-5d632f0c65be-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.718910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.718944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-root\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.718972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.718996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-wtmp\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719055 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxtx\" (UniqueName: \"kubernetes.io/projected/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-kube-api-access-sdxtx\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-textfile\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wss\" (UniqueName: \"kubernetes.io/projected/7799ffba-3b3f-456a-ac76-5d632f0c65be-kube-api-access-47wss\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-sys\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.719356 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.719305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-metrics-client-ca\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.786098 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.786070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.819832 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.819995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-root\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.819995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.819995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.819995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-wtmp\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.819995 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.819937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-root\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-wtmp\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxtx\" (UniqueName: \"kubernetes.io/projected/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-kube-api-access-sdxtx\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-textfile\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wss\" (UniqueName: \"kubernetes.io/projected/7799ffba-3b3f-456a-ac76-5d632f0c65be-kube-api-access-47wss\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820221 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-sys\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-metrics-client-ca\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7799ffba-3b3f-456a-ac76-5d632f0c65be-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:46.820207    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:46.820449    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls podName:d34fa22a-dfcd-4422-b4af-3bcfdaafea0d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:47.320429689 +0000 UTC m=+138.644751090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls") pod "node-exporter-8jbj9" (UID: "d34fa22a-dfcd-4422-b4af-3bcfdaafea0d") : secret "node-exporter-tls" not found
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-textfile\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820554 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820909 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-sys\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.820962 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.820936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7799ffba-3b3f-456a-ac76-5d632f0c65be-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.821056 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.821031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-metrics-client-ca\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.822387 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.822365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.822481 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.822437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7799ffba-3b3f-456a-ac76-5d632f0c65be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.822557 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.822540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.849410 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.849386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wss\" (UniqueName: \"kubernetes.io/projected/7799ffba-3b3f-456a-ac76-5d632f0c65be-kube-api-access-47wss\") pod \"openshift-state-metrics-86c5bc79d6-7vhjx\" (UID: \"7799ffba-3b3f-456a-ac76-5d632f0c65be\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:46.851182 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.851161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxtx\" (UniqueName: \"kubernetes.io/projected/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-kube-api-access-sdxtx\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:46.874180 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:46.874158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"
Sep 17 17:58:47.072398 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.072369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:47.202229 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.202204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx"]
Sep 17 17:58:47.203278 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:47.203255    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7799ffba_3b3f_456a_ac76_5d632f0c65be.slice/crio-5ce9987ae3a48dbbcf21b80373991bde2aa16eb0b7cb524125711cadd3bfa0eb WatchSource:0}: Error finding container 5ce9987ae3a48dbbcf21b80373991bde2aa16eb0b7cb524125711cadd3bfa0eb: Status 404 returned error can't find the container with id 5ce9987ae3a48dbbcf21b80373991bde2aa16eb0b7cb524125711cadd3bfa0eb
Sep 17 17:58:47.324796 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.324743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:47.326947 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.326926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d34fa22a-dfcd-4422-b4af-3bcfdaafea0d-node-exporter-tls\") pod \"node-exporter-8jbj9\" (UID: \"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d\") " pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:47.470269 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.470185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:47.520547 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.520515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx" event={"ID":"7799ffba-3b3f-456a-ac76-5d632f0c65be","Type":"ContainerStarted","Data":"3dd77563bfb64db33ec815c1b2476f3d87041bdfacdc4688332fb0030cdbeede"}
Sep 17 17:58:47.520669 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.520573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx" event={"ID":"7799ffba-3b3f-456a-ac76-5d632f0c65be","Type":"ContainerStarted","Data":"805b472fbff6dd2e955c3c4844650541118a0973890d23d028f4722d35b8d4c5"}
Sep 17 17:58:47.520669 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.520593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx" event={"ID":"7799ffba-3b3f-456a-ac76-5d632f0c65be","Type":"ContainerStarted","Data":"5ce9987ae3a48dbbcf21b80373991bde2aa16eb0b7cb524125711cadd3bfa0eb"}
Sep 17 17:58:47.545867 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.545840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jbj9"
Sep 17 17:58:47.554672 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:47.554643    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34fa22a_dfcd_4422_b4af_3bcfdaafea0d.slice/crio-739e77f2d9f44baef3d36dd6706ab908293b91181d719f3062293e08a7648bdc WatchSource:0}: Error finding container 739e77f2d9f44baef3d36dd6706ab908293b91181d719f3062293e08a7648bdc: Status 404 returned error can't find the container with id 739e77f2d9f44baef3d36dd6706ab908293b91181d719f3062293e08a7648bdc
Sep 17 17:58:47.621989 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.621965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:47.801330 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.801263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:47.980321 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:47.980282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.070304 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.070228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:48.073384 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.073363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.078000 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.077920    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:58:48.078192 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:58:48.078192 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:58:48.078370 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:58:48.078565 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078549    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hz9nx\""
Sep 17 17:58:48.078565 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:58:48.078830 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.078580    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:58:48.079031 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.079011    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:58:48.080256 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.080236    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:58:48.080823 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.080805    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:58:48.107868 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.107778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:48.138745 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.138715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.233696 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233696 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2cs\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.233953 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.233974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.234004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.234026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.234104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.234134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.234205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.234161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.330554 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.330464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.335542 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335640 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335758 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335887 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335887 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335997 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335997 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.335997 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.335980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336132 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336132 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336132 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336132 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336333 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2cs\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.336446 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.337039 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.337039 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.336832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340424 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.339961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340424 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.340140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340424 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.340161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340424 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.340378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340683 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.340428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.340683 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.340541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.341176 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.341156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.341628 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.341593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.341815 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.341789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.353711 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.353689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2cs\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs\") pod \"alertmanager-main-0\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.386664 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.386641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.527248 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.526807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbj9" event={"ID":"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d","Type":"ContainerStarted","Data":"739e77f2d9f44baef3d36dd6706ab908293b91181d719f3062293e08a7648bdc"}
Sep 17 17:58:48.785464 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:48.785433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:48.788316 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:48.788291    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78717fc3_d2e8_481f_a549_e30f7472f536.slice/crio-d16863db9ef7c76f30ce7b5622242824ec432c992c0a097d34cd36cf8d560ebc WatchSource:0}: Error finding container d16863db9ef7c76f30ce7b5622242824ec432c992c0a097d34cd36cf8d560ebc: Status 404 returned error can't find the container with id d16863db9ef7c76f30ce7b5622242824ec432c992c0a097d34cd36cf8d560ebc
Sep 17 17:58:49.532582 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:49.532550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx" event={"ID":"7799ffba-3b3f-456a-ac76-5d632f0c65be","Type":"ContainerStarted","Data":"d01814dfccfeea23a95ccb8366e6adaca2f98b0712d8e140a7f9327901c3847b"}
Sep 17 17:58:49.534302 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:49.534271    2577 generic.go:356] "Generic (PLEG): container finished" podID="d34fa22a-dfcd-4422-b4af-3bcfdaafea0d" containerID="d736ad0911c3a599413caf43bb248f99f1971e3f590f0e6a09ec9d4c8c84d6b3" exitCode=0
Sep 17 17:58:49.534410 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:49.534320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbj9" event={"ID":"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d","Type":"ContainerDied","Data":"d736ad0911c3a599413caf43bb248f99f1971e3f590f0e6a09ec9d4c8c84d6b3"}
Sep 17 17:58:49.535652 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:49.535633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"d16863db9ef7c76f30ce7b5622242824ec432c992c0a097d34cd36cf8d560ebc"}
Sep 17 17:58:49.582141 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:49.582090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7vhjx" podStartSLOduration=2.39912531 podStartE2EDuration="3.582068691s" totalImagesPullingTime="1.182943381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:49.580206048 +0000 UTC m=+140.904527472" watchObservedRunningTime="2026-09-17 17:58:49.582068691 +0000 UTC m=+140.906390118"
Sep 17 17:58:50.011893 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.011854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.036171 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.036145    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"]
Sep 17 17:58:50.039638 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.039623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.052407 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.052362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:58:50.052486 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.052437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:58:50.053082 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.053069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:58:50.055913 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:50.055892    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret"
Sep 17 17:58:50.056090 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.056077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ekr4v5eqfonu7\""
Sep 17 17:58:50.058983 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:58:50.058955    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-dockercfg-d6fv2\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d6fv2\"" type="*v1.Secret"
Sep 17 17:58:50.065792 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.065774    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:58:50.119580 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.119553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"]
Sep 17 17:58:50.151853 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.151823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.151975 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.151879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.151975 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.151930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jzk\" (UniqueName: \"kubernetes.io/projected/960d49a8-0691-4460-80e5-0064307f92ca-kube-api-access-x5jzk\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.151975 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.151965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.152100 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.152016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-grpc-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.152137 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.152095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.152137 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.152125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.152216 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.152151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960d49a8-0691-4460-80e5-0064307f92ca-metrics-client-ca\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.170320 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.170294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.253110 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253241 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-grpc-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253241 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253241 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253241 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960d49a8-0691-4460-80e5-0064307f92ca-metrics-client-ca\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253241 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253473 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.253473 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.253265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jzk\" (UniqueName: \"kubernetes.io/projected/960d49a8-0691-4460-80e5-0064307f92ca-kube-api-access-x5jzk\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.254060 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.254002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960d49a8-0691-4460-80e5-0064307f92ca-metrics-client-ca\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.255836 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.255803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.255923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.255810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-grpc-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.255923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.255818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.255990 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.255966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.256135 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.256120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-tls\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.283335 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.283312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jzk\" (UniqueName: \"kubernetes.io/projected/960d49a8-0691-4460-80e5-0064307f92ca-kube-api-access-x5jzk\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:50.323301 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.323248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.491009 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.490982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.541425 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.541393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbj9" event={"ID":"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d","Type":"ContainerStarted","Data":"9eb2a3bc28474570359f15e73ea681c66db59c16d46a8b62690c5cb69babacf5"}
Sep 17 17:58:50.541548 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.541447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbj9" event={"ID":"d34fa22a-dfcd-4422-b4af-3bcfdaafea0d","Type":"ContainerStarted","Data":"46275f54d593c2ef0238d83af5816617db3d498e1760793ad1578d8cb1e62ab4"}
Sep 17 17:58:50.542906 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.542885    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed" exitCode=0
Sep 17 17:58:50.543005 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.542972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"}
Sep 17 17:58:50.576295 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.576225    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8jbj9" podStartSLOduration=3.55588152 podStartE2EDuration="4.576211541s" totalImagesPullingTime="958.496227ms" totalInitContainerRuntime="61.833794ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:50.572605105 +0000 UTC m=+141.896926528" watchObservedRunningTime="2026-09-17 17:58:50.576211541 +0000 UTC m=+141.900532963"
Sep 17 17:58:50.684589 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.684566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.937209 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.937173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:58:50.946002 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:50.945963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/960d49a8-0691-4460-80e5-0064307f92ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dd797f6c-h5sf9\" (UID: \"960d49a8-0691-4460-80e5-0064307f92ca\") " pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:51.081146 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.081113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:51.148298 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.148268    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b9c9c9954-hxm49"]
Sep 17 17:58:51.151207 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.151193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.161332 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.161313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a5uq5t21l45m0\""
Sep 17 17:58:51.161332 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.161321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:58:51.161533 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.161521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:58:51.162093 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.162078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:58:51.162162 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.162081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zdbsb\""
Sep 17 17:58:51.163037 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.163024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:58:51.185564 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.185545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b9c9c9954-hxm49"]
Sep 17 17:58:51.261961 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.261872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpd4z\" (UniqueName: \"kubernetes.io/projected/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-kube-api-access-vpd4z\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.261961 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.261926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-client-certs\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.262148 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.261996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-audit-log\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.262148 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.262055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.262148 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.262081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-client-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.262265 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.262157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-metrics-server-audit-profiles\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.262265 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.262190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-tls\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.360154 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.360122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:51.363333 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-audit-log\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-client-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-metrics-server-audit-profiles\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363585 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-tls\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363585 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpd4z\" (UniqueName: \"kubernetes.io/projected/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-kube-api-access-vpd4z\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363585 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-client-certs\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.363882 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.363738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-audit-log\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.364188 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.364163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.364535 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.364510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-metrics-server-audit-profiles\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.366312 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.366265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-tls\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.366466 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.366440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-client-ca-bundle\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.366579 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.366520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-secret-metrics-server-client-certs\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.373935 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.373918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpd4z\" (UniqueName: \"kubernetes.io/projected/0508cfb5-7aa5-4d89-9f15-62f3f15c30be-kube-api-access-vpd4z\") pod \"metrics-server-6b9c9c9954-hxm49\" (UID: \"0508cfb5-7aa5-4d89-9f15-62f3f15c30be\") " pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.460462 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.460425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:58:51.580412 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.580332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d6fv2\""
Sep 17 17:58:51.588032 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:51.588007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:52.033901 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.033877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b9c9c9954-hxm49"]
Sep 17 17:58:52.052748 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:52.052719    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0508cfb5_7aa5_4d89_9f15_62f3f15c30be.slice/crio-7ae76f9398f8ceaab8cac05fa46565a49cb4f15a7fb4cd7d0884210bdf24df9e WatchSource:0}: Error finding container 7ae76f9398f8ceaab8cac05fa46565a49cb4f15a7fb4cd7d0884210bdf24df9e: Status 404 returned error can't find the container with id 7ae76f9398f8ceaab8cac05fa46565a49cb4f15a7fb4cd7d0884210bdf24df9e
Sep 17 17:58:52.053823 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:52.053800    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960d49a8_0691_4460_80e5_0064307f92ca.slice/crio-ed7174864ecb8a4e3fc6d407bcd04c0ad0c3c8aeeb4c23fa15bf130eb8b00bae WatchSource:0}: Error finding container ed7174864ecb8a4e3fc6d407bcd04c0ad0c3c8aeeb4c23fa15bf130eb8b00bae: Status 404 returned error can't find the container with id ed7174864ecb8a4e3fc6d407bcd04c0ad0c3c8aeeb4c23fa15bf130eb8b00bae
Sep 17 17:58:52.059932 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.059910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"]
Sep 17 17:58:52.555870 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.555787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"ed7174864ecb8a4e3fc6d407bcd04c0ad0c3c8aeeb4c23fa15bf130eb8b00bae"}
Sep 17 17:58:52.560052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.560020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"}
Sep 17 17:58:52.560176 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.560096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"}
Sep 17 17:58:52.560176 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.560137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"}
Sep 17 17:58:52.560322 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.560176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"}
Sep 17 17:58:52.560322 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.560207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"}
Sep 17 17:58:52.561605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:52.561575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49" event={"ID":"0508cfb5-7aa5-4d89-9f15-62f3f15c30be","Type":"ContainerStarted","Data":"7ae76f9398f8ceaab8cac05fa46565a49cb4f15a7fb4cd7d0884210bdf24df9e"}
Sep 17 17:58:53.469931 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.469895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:53.476247 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.475670    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:53.480072 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.480055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.488886 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.488822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:58:53.489100 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.489081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:58:53.489205 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.489135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:58:53.490149 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:58:53.490149 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490042    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f96rp\""
Sep 17 17:58:53.490149 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490058    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:58:53.490353 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490042    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:58:53.490583 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8l2995kr12md1\""
Sep 17 17:58:53.490642 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.490545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:58:53.492122 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.492105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:58:53.506497 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.506464    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:58:53.517460 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.517440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:53.582551 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.582551 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzfd\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.582989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.583443 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.583274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684506 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684693 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684693 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684693 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684890 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684890 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.684890 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685038 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685038 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685038 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.684953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685038 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzfd\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685233 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685516 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685516 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.685516 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.685298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.686285 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.686241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.686971 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.686922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.687691 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.687668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.688145 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.688111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.688639 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.688329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.688639 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.688383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.688800 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.688740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.688937 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.688914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.689394 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.689369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.689481 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.689434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.689605 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.689580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.689840 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.689820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.689970 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.689942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.690361 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.690335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.691130 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.691103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.691346 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.691288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.692324 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.692292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.692324 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.692312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.692476 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.692428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.715671 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.715645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzfd\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd\") pod \"prometheus-k8s-0\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:53.789531 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:53.789451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:54.342229 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.342205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:54.351828 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:58:54.351803    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb5b275_3c8a_40b4_8034_01dc2ed76564.slice/crio-5b1d06e5884590e52832863530e7a667513de65e25b9dc2ef2f72ace08effecd WatchSource:0}: Error finding container 5b1d06e5884590e52832863530e7a667513de65e25b9dc2ef2f72ace08effecd: Status 404 returned error can't find the container with id 5b1d06e5884590e52832863530e7a667513de65e25b9dc2ef2f72ace08effecd
Sep 17 17:58:54.573780 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.573689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerStarted","Data":"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"}
Sep 17 17:58:54.575379 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.575355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49" event={"ID":"0508cfb5-7aa5-4d89-9f15-62f3f15c30be","Type":"ContainerStarted","Data":"5e48a954b5769cbcad723aaea222a52d5127c689795027600596fa9c5c920ca3"}
Sep 17 17:58:54.576871 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.576847    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f" exitCode=0
Sep 17 17:58:54.576980 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.576931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"}
Sep 17 17:58:54.577034 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.577004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"5b1d06e5884590e52832863530e7a667513de65e25b9dc2ef2f72ace08effecd"}
Sep 17 17:58:54.580384 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"e015b72c193dac93cdd944c98141e7cdd2e85fac6b7a0e38ff9ae2893d489d68"}
Sep 17 17:58:54.580481 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"27131c0524b97005b8742ad8a252f8ac82435ff3cf8847cb10e855aa7cacdba6"}
Sep 17 17:58:54.580481 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"fd8656bb8a11ee573dfe2684e0fabd9ff8ef362a331a53654e7cfa95e877b67f"}
Sep 17 17:58:54.580559 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:58:54.580559 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"f022509bb9115071cd5c7aa399495467f614adad86e5e4f9f9f367ba2a43cf2d"}
Sep 17 17:58:54.580559 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"9528a25589a5fe0d7fecfe1ddd7e452053f843a6b7ff89bde8670fe77ec12acb"}
Sep 17 17:58:54.580639 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.580574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" event={"ID":"960d49a8-0691-4460-80e5-0064307f92ca","Type":"ContainerStarted","Data":"09f6a872990ccbb2f3c053c91dd5c72aa0a131027e5201497a05e71b720cc15c"}
Sep 17 17:58:54.605427 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.605380    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.399350128 podStartE2EDuration="6.605361074s" totalImagesPullingTime="4.186019631s" totalInitContainerRuntime="19.991315ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:54.603408984 +0000 UTC m=+145.927730407" watchObservedRunningTime="2026-09-17 17:58:54.605361074 +0000 UTC m=+145.929682501"
Sep 17 17:58:54.637335 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.637280    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9" podStartSLOduration=2.641106212 podStartE2EDuration="4.63726275s" totalImagesPullingTime="1.996156538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:54.634946644 +0000 UTC m=+145.959268092" watchObservedRunningTime="2026-09-17 17:58:54.63726275 +0000 UTC m=+145.961584175"
Sep 17 17:58:54.663843 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:54.663801    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49" podStartSLOduration=1.668496811 podStartE2EDuration="3.663788016s" totalImagesPullingTime="1.995291205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:54.662576647 +0000 UTC m=+145.986898071" watchObservedRunningTime="2026-09-17 17:58:54.663788016 +0000 UTC m=+145.988109439"
Sep 17 17:58:55.879846 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:55.879802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:55.983024 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:55.982983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:57.598701 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:57.598665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"}
Sep 17 17:58:57.599090 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:57.598779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"}
Sep 17 17:58:58.605971 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.605931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"}
Sep 17 17:58:58.606451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.606050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"}
Sep 17 17:58:58.606451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.606099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"}
Sep 17 17:58:58.606451 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.606142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerStarted","Data":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"}
Sep 17 17:58:58.790386 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.790352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:58.847562 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.847499    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.98159933 podStartE2EDuration="5.847478211s" totalImagesPullingTime="2.845681059s" totalInitContainerRuntime="20.197822ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:58.843189366 +0000 UTC m=+150.167510790" watchObservedRunningTime="2026-09-17 17:58:58.847478211 +0000 UTC m=+150.171799634"
Sep 17 17:58:58.864097 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:58.864025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.071141 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:59.071108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.180452 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:59.180377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.282746 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:59.282713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.752031 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:58:59.751999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.073245 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.073163    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5455578cf6-b66qs"]
Sep 17 17:59:00.073515 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:59:00.073488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5455578cf6-b66qs" podUID="1f878402-cbf9-44e8-82d2-3b47b25244c5"
Sep 17 17:59:00.147102 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.147074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.249917 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.249883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.350918 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.350840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.449789 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.449737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.590959 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.590935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6dd797f6c-h5sf9"
Sep 17 17:59:00.618636 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.618606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:59:00.627600 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.627572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:59:00.647905 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.647877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.647913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdr5z\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.647949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648052 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.647995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648207 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.648135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:00.648207 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.648152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648207 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.648196    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648359 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.648247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets\") pod \"1f878402-cbf9-44e8-82d2-3b47b25244c5\" (UID: \"1f878402-cbf9-44e8-82d2-3b47b25244c5\") "
Sep 17 17:59:00.648522 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.648500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:00.649344 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.649320    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-trusted-ca\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.649455 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.649351    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f878402-cbf9-44e8-82d2-3b47b25244c5-ca-trust-extracted\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.652114 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.652088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:00.652603 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.652576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:00.652698 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.652643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:00.652880 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.652846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:00.652989 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.652970    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z" (OuterVolumeSpecName: "kube-api-access-tdr5z") pod "1f878402-cbf9-44e8-82d2-3b47b25244c5" (UID: "1f878402-cbf9-44e8-82d2-3b47b25244c5"). InnerVolumeSpecName "kube-api-access-tdr5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:00.749773 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.749714    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-image-registry-private-configuration\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.749773 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.749747    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-certificates\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.749959 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.749782    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f878402-cbf9-44e8-82d2-3b47b25244c5-installation-pull-secrets\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.749959 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.749795    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr5z\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-kube-api-access-tdr5z\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:00.749959 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:00.749806    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-bound-sa-token\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:01.622802 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:01.622772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5455578cf6-b66qs"
Sep 17 17:59:01.684650 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:01.684618    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5455578cf6-b66qs"]
Sep 17 17:59:01.691050 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:01.691025    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5455578cf6-b66qs"]
Sep 17 17:59:01.759418 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:01.759383    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f878402-cbf9-44e8-82d2-3b47b25244c5-registry-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.352768 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:02.352723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:02.442277 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:02.442244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:02.941903 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:02.941869    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f878402-cbf9-44e8-82d2-3b47b25244c5" path="/var/lib/kubelet/pods/1f878402-cbf9-44e8-82d2-3b47b25244c5/volumes"
Sep 17 17:59:06.795105 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:06.795072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.087969 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:07.087892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.261579 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:07.261547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.383130 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:07.383098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.481902 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:07.481866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.561077 ip-10-0-133-151 kubenswrapper[2577]: E0917 17:59:07.561034    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r9f4z" podUID="c061ac52-ce91-447d-86da-66497b456a39"
Sep 17 17:59:07.646999 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:07.646931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:09.359620 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:09.359595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:09.479893 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:09.479860    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:11.460961 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:11.460924    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:59:11.461429 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:11.461006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:59:12.450671 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:12.450595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:12.452940 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:12.452916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c061ac52-ce91-447d-86da-66497b456a39-metrics-tls\") pod \"dns-default-r9f4z\" (UID: \"c061ac52-ce91-447d-86da-66497b456a39\") " pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:12.750744 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:12.750665    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5wl9d\""
Sep 17 17:59:12.758161 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:12.758141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:13.040437 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:13.040412    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9f4z"]
Sep 17 17:59:13.042810 ip-10-0-133-151 kubenswrapper[2577]: W0917 17:59:13.042784    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc061ac52_ce91_447d_86da_66497b456a39.slice/crio-4f773872cdb7ddc301cd9f688a8781102e6d8f81717dcdd189db8ebf75679c45 WatchSource:0}: Error finding container 4f773872cdb7ddc301cd9f688a8781102e6d8f81717dcdd189db8ebf75679c45: Status 404 returned error can't find the container with id 4f773872cdb7ddc301cd9f688a8781102e6d8f81717dcdd189db8ebf75679c45
Sep 17 17:59:13.672923 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:13.672889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9f4z" event={"ID":"c061ac52-ce91-447d-86da-66497b456a39","Type":"ContainerStarted","Data":"4f773872cdb7ddc301cd9f688a8781102e6d8f81717dcdd189db8ebf75679c45"}
Sep 17 17:59:14.734823 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:14.734785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:14.829452 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:14.829426    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:15.632109 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:15.632083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxjw4_3ed57787-1808-4845-a93a-d2ddea2000cd/dns-node-resolver/0.log"
Sep 17 17:59:15.683258 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:15.683226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9f4z" event={"ID":"c061ac52-ce91-447d-86da-66497b456a39","Type":"ContainerStarted","Data":"8aec665a9691078ff2557f18303dc6e26c07342a78502c6e086ac64a89f2e056"}
Sep 17 17:59:15.683412 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:15.683276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9f4z" event={"ID":"c061ac52-ce91-447d-86da-66497b456a39","Type":"ContainerStarted","Data":"034607c85c03c9b339b9a56304846ddbb988d6c5d4c2e3fc7bab239ad5ed0b3e"}
Sep 17 17:59:15.683412 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:15.683318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:15.720594 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:15.720550    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9f4z" podStartSLOduration=129.940882667 podStartE2EDuration="2m11.720536271s" totalImagesPullingTime="1.779653604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:15.719706844 +0000 UTC m=+167.044028267" watchObservedRunningTime="2026-09-17 17:59:15.720536271 +0000 UTC m=+167.044857694"
Sep 17 17:59:16.345275 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:16.345239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:21.054715 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:21.054685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:21.209037 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:21.209011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:25.689293 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:25.689266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9f4z"
Sep 17 17:59:31.466373 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:31.466343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:59:31.470040 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:31.470015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b9c9c9954-hxm49"
Sep 17 17:59:31.964535 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:31.964504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:32.071183 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:32.071137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:32.955026 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:32.954993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:33.062781 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:33.062723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:33.957711 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:33.957676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.151150 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:34.151115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.893283 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:34.893249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.983027 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:34.982995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:36.781275 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:36.781240    2577 generic.go:356] "Generic (PLEG): container finished" podID="b43082c0-82d8-4e90-873e-17041bfaf5cd" containerID="c70263fa19a094e438fe511c0755de05142ee5228b1c827059cc9df29a96a940" exitCode=0
Sep 17 17:59:36.781777 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:36.781296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc" event={"ID":"b43082c0-82d8-4e90-873e-17041bfaf5cd","Type":"ContainerDied","Data":"c70263fa19a094e438fe511c0755de05142ee5228b1c827059cc9df29a96a940"}
Sep 17 17:59:36.781777 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:36.781586    2577 scope.go:117] "RemoveContainer" containerID="c70263fa19a094e438fe511c0755de05142ee5228b1c827059cc9df29a96a940"
Sep 17 17:59:37.787305 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:37.787270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-h9nfc" event={"ID":"b43082c0-82d8-4e90-873e-17041bfaf5cd","Type":"ContainerStarted","Data":"82bf2e27efddad2ff8d1d981c58abf8229ecce78d8c296bf9bf075b5e727b46e"}
Sep 17 17:59:37.873279 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:37.873251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:43.902619 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:43.902589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:43.979454 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:43.979424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:46.473972 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:46.473933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:46.559960 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:46.559929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:53.790131 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:53.790097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:59:53.810336 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:53.810314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:59:53.866892 ip-10-0-133-151 kubenswrapper[2577]: I0917 17:59:53.866867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:06.391770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:06.391713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:06.754501 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:06.754417    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:07.523459 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.523417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:07.523970 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.523793    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="alertmanager" containerID="cri-o://4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb" gracePeriod=120
Sep 17 18:00:07.523970 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.523881    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-metric" containerID="cri-o://ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b" gracePeriod=120
Sep 17 18:00:07.524078 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.524024    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-web" containerID="cri-o://3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be" gracePeriod=120
Sep 17 18:00:07.524078 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.524042    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy" containerID="cri-o://ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296" gracePeriod=120
Sep 17 18:00:07.524159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.524020    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="prom-label-proxy" containerID="cri-o://104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e" gracePeriod=120
Sep 17 18:00:07.524196 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.524159    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="config-reloader" containerID="cri-o://6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283" gracePeriod=120
Sep 17 18:00:07.910408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910376    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e" exitCode=0
Sep 17 18:00:07.910408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910401    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296" exitCode=0
Sep 17 18:00:07.910408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910409    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283" exitCode=0
Sep 17 18:00:07.910408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910415    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb" exitCode=0
Sep 17 18:00:07.910696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"}
Sep 17 18:00:07.910696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"}
Sep 17 18:00:07.910696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"}
Sep 17 18:00:07.910696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:07.910557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"}
Sep 17 18:00:08.776859 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.776835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:08.899090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2cs\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899778 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899778 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db\") pod \"78717fc3-d2e8-481f-a549-e30f7472f536\" (UID: \"78717fc3-d2e8-481f-a549-e30f7472f536\") "
Sep 17 18:00:08.899778 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.899712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:08.900257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.900184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:08.900694 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.900666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:08.901858 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.901822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume" (OuterVolumeSpecName: "config-volume") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.902340 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.902315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs" (OuterVolumeSpecName: "kube-api-access-9s2cs") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "kube-api-access-9s2cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:08.902679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.902631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.902900 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.902864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out" (OuterVolumeSpecName: "config-out") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:08.903199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.903162    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.903297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.903215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.903297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.903256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.903404 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.903349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:08.906899 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.906705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.913189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.913167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config" (OuterVolumeSpecName: "web-config") pod "78717fc3-d2e8-481f-a549-e30f7472f536" (UID: "78717fc3-d2e8-481f-a549-e30f7472f536"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:08.917334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917308    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b" exitCode=0
Sep 17 18:00:08.917334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917333    2577 generic.go:356] "Generic (PLEG): container finished" podID="78717fc3-d2e8-481f-a549-e30f7472f536" containerID="3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be" exitCode=0
Sep 17 18:00:08.917437 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"}
Sep 17 18:00:08.917437 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:08.917503 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"}
Sep 17 18:00:08.917503 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78717fc3-d2e8-481f-a549-e30f7472f536","Type":"ContainerDied","Data":"d16863db9ef7c76f30ce7b5622242824ec432c992c0a097d34cd36cf8d560ebc"}
Sep 17 18:00:08.917560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.917514    2577 scope.go:117] "RemoveContainer" containerID="104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"
Sep 17 18:00:08.926005 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.925985    2577 scope.go:117] "RemoveContainer" containerID="ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"
Sep 17 18:00:08.933244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.933229    2577 scope.go:117] "RemoveContainer" containerID="ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"
Sep 17 18:00:08.940222 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.940207    2577 scope.go:117] "RemoveContainer" containerID="3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"
Sep 17 18:00:08.945117 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.945094    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:08.947529 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.947512    2577 scope.go:117] "RemoveContainer" containerID="6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"
Sep 17 18:00:08.954265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.954232    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:08.959601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.959584    2577 scope.go:117] "RemoveContainer" containerID="4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"
Sep 17 18:00:08.967059 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.967041    2577 scope.go:117] "RemoveContainer" containerID="c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"
Sep 17 18:00:08.973555 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.973539    2577 scope.go:117] "RemoveContainer" containerID="104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"
Sep 17 18:00:08.973827 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.973808    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e\": container with ID starting with 104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e not found: ID does not exist" containerID="104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"
Sep 17 18:00:08.973914 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.973833    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"} err="failed to get container status \"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e\": rpc error: code = NotFound desc = could not find container \"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e\": container with ID starting with 104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e not found: ID does not exist"
Sep 17 18:00:08.973914 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.973864    2577 scope.go:117] "RemoveContainer" containerID="ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"
Sep 17 18:00:08.974092 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.974074    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b\": container with ID starting with ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b not found: ID does not exist" containerID="ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"
Sep 17 18:00:08.974142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974100    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"} err="failed to get container status \"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b\": rpc error: code = NotFound desc = could not find container \"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b\": container with ID starting with ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b not found: ID does not exist"
Sep 17 18:00:08.974142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974115    2577 scope.go:117] "RemoveContainer" containerID="ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"
Sep 17 18:00:08.974416 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.974393    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296\": container with ID starting with ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296 not found: ID does not exist" containerID="ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"
Sep 17 18:00:08.974465 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974425    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"} err="failed to get container status \"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296\": rpc error: code = NotFound desc = could not find container \"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296\": container with ID starting with ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296 not found: ID does not exist"
Sep 17 18:00:08.974465 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974439    2577 scope.go:117] "RemoveContainer" containerID="3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"
Sep 17 18:00:08.974680 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.974664    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be\": container with ID starting with 3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be not found: ID does not exist" containerID="3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"
Sep 17 18:00:08.974722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974683    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"} err="failed to get container status \"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be\": rpc error: code = NotFound desc = could not find container \"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be\": container with ID starting with 3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be not found: ID does not exist"
Sep 17 18:00:08.974722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974695    2577 scope.go:117] "RemoveContainer" containerID="6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"
Sep 17 18:00:08.974961 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.974946    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283\": container with ID starting with 6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283 not found: ID does not exist" containerID="6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"
Sep 17 18:00:08.975007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974965    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"} err="failed to get container status \"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283\": rpc error: code = NotFound desc = could not find container \"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283\": container with ID starting with 6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283 not found: ID does not exist"
Sep 17 18:00:08.975007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.974977    2577 scope.go:117] "RemoveContainer" containerID="4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"
Sep 17 18:00:08.975220 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.975201    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb\": container with ID starting with 4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb not found: ID does not exist" containerID="4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"
Sep 17 18:00:08.975284 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975227    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"} err="failed to get container status \"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb\": rpc error: code = NotFound desc = could not find container \"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb\": container with ID starting with 4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb not found: ID does not exist"
Sep 17 18:00:08.975284 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975263    2577 scope.go:117] "RemoveContainer" containerID="c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"
Sep 17 18:00:08.975529 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:08.975513    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed\": container with ID starting with c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed not found: ID does not exist" containerID="c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"
Sep 17 18:00:08.975573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975533    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"} err="failed to get container status \"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed\": rpc error: code = NotFound desc = could not find container \"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed\": container with ID starting with c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed not found: ID does not exist"
Sep 17 18:00:08.975573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975545    2577 scope.go:117] "RemoveContainer" containerID="104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"
Sep 17 18:00:08.975785 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975763    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e"} err="failed to get container status \"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e\": rpc error: code = NotFound desc = could not find container \"104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e\": container with ID starting with 104d967f6bb448572c413405d1e70f8b2a7102eaaf922f40b166cee965aed44e not found: ID does not exist"
Sep 17 18:00:08.975836 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.975787    2577 scope.go:117] "RemoveContainer" containerID="ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"
Sep 17 18:00:08.976023 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976005    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b"} err="failed to get container status \"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b\": rpc error: code = NotFound desc = could not find container \"ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b\": container with ID starting with ae0dd6900dcc16d0263f9a45ba8cf5ffbd4d0955742b958459e963a6b572a56b not found: ID does not exist"
Sep 17 18:00:08.976083 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976026    2577 scope.go:117] "RemoveContainer" containerID="ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"
Sep 17 18:00:08.976263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976247    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296"} err="failed to get container status \"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296\": rpc error: code = NotFound desc = could not find container \"ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296\": container with ID starting with ff89cebea9254da3b6558ecec8638dbfa1c5f61a514b38eef7eb178901b92296 not found: ID does not exist"
Sep 17 18:00:08.976333 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976264    2577 scope.go:117] "RemoveContainer" containerID="3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"
Sep 17 18:00:08.976500 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976483    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be"} err="failed to get container status \"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be\": rpc error: code = NotFound desc = could not find container \"3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be\": container with ID starting with 3f01fad1924836c276a6245c5a6045ba4f276ce8c466384f23e2f6a2312732be not found: ID does not exist"
Sep 17 18:00:08.976540 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976501    2577 scope.go:117] "RemoveContainer" containerID="6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"
Sep 17 18:00:08.976745 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976729    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283"} err="failed to get container status \"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283\": rpc error: code = NotFound desc = could not find container \"6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283\": container with ID starting with 6dfa80305fb8847c08ea301a1117ad77c13aa06aba65a9b06735d6acd90f8283 not found: ID does not exist"
Sep 17 18:00:08.976852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976745    2577 scope.go:117] "RemoveContainer" containerID="4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"
Sep 17 18:00:08.976954 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976938    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb"} err="failed to get container status \"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb\": rpc error: code = NotFound desc = could not find container \"4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb\": container with ID starting with 4204dd314e3f448a4e1c66890bb84d88f310f5c10d32177669af2d410ae294fb not found: ID does not exist"
Sep 17 18:00:08.976998 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.976954    2577 scope.go:117] "RemoveContainer" containerID="c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"
Sep 17 18:00:08.977180 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.977161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed"} err="failed to get container status \"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed\": rpc error: code = NotFound desc = could not find container \"c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed\": container with ID starting with c2934551885575ddd8b280481ef670b2d5c81d5b9aefe81b4faf4a85b2ded4ed not found: ID does not exist"
Sep 17 18:00:08.995872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.995851    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:08.996169 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="config-reloader"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996171    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="config-reloader"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996188    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-web"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996201    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-web"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996209    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="alertmanager"
Sep 17 18:00:08.996215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996214    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="alertmanager"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996226    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996243    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="init-config-reloader"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996249    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="init-config-reloader"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996254    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="prom-label-proxy"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996281    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="prom-label-proxy"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996329    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="config-reloader"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996337    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="alertmanager"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996344    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-web"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996349    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="prom-label-proxy"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996354    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:08.996411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.996360    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" containerName="kube-rbac-proxy"
Sep 17 18:00:08.999635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:08.999622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.000187 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000159    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000187 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000185    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-cluster-tls-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000195    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-main-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000204    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-tls-assets\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000212    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000221    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78717fc3-d2e8-481f-a549-e30f7472f536-metrics-client-ca\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000229    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-web-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000238    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000245    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-config-out\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000253    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78717fc3-d2e8-481f-a549-e30f7472f536-alertmanager-main-db\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000261    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000269    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78717fc3-d2e8-481f-a549-e30f7472f536-config-volume\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.000335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.000276    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2cs\" (UniqueName: \"kubernetes.io/projected/78717fc3-d2e8-481f-a549-e30f7472f536-kube-api-access-9s2cs\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.002746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.002723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:00:09.003004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.002989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:00:09.003110 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:00:09.003242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hz9nx\""
Sep 17 18:00:09.003314 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:00:09.003580 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003565    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:00:09.003805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:00:09.003930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.003916    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:00:09.004195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.004180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:00:09.010093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.010074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:00:09.020489 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.020467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.101128 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101128 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-kube-api-access-8r2h5\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-web-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-config-out\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.101659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.101508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.202774 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.202774 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.202774 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-kube-api-access-8r2h5\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-web-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-config-out\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.202972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.203016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.203060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.203079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.203331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.203818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.203792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.204218 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.204194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dffac96-c450-4e20-a67a-0839b65791c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.205900 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.205876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dffac96-c450-4e20-a67a-0839b65791c5-config-out\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.205961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206197 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206307 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-web-config\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.206713 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.206693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0dffac96-c450-4e20-a67a-0839b65791c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.207629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.207613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.213306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.213287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/0dffac96-c450-4e20-a67a-0839b65791c5-kube-api-access-8r2h5\") pod \"alertmanager-main-0\" (UID: \"0dffac96-c450-4e20-a67a-0839b65791c5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.309141 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.309117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.591040 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.591017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.593413 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:00:09.593374    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dffac96_c450_4e20_a67a_0839b65791c5.slice/crio-c27e9398a4dfb21d1aa0031de52ab02ef3ddee56fcca1ab507960039664040f9 WatchSource:0}: Error finding container c27e9398a4dfb21d1aa0031de52ab02ef3ddee56fcca1ab507960039664040f9: Status 404 returned error can't find the container with id c27e9398a4dfb21d1aa0031de52ab02ef3ddee56fcca1ab507960039664040f9
Sep 17 18:00:09.923932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.923902    2577 generic.go:356] "Generic (PLEG): container finished" podID="0dffac96-c450-4e20-a67a-0839b65791c5" containerID="8e2b485f0ca9fa9f5bcfadd05b5481454c0c9e2110669681b8102c7d5559e3a7" exitCode=0
Sep 17 18:00:09.924319 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.923987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerDied","Data":"8e2b485f0ca9fa9f5bcfadd05b5481454c0c9e2110669681b8102c7d5559e3a7"}
Sep 17 18:00:09.924319 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:09.924051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"c27e9398a4dfb21d1aa0031de52ab02ef3ddee56fcca1ab507960039664040f9"}
Sep 17 18:00:10.941934 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.941900    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78717fc3-d2e8-481f-a549-e30f7472f536" path="/var/lib/kubelet/pods/78717fc3-d2e8-481f-a549-e30f7472f536/volumes"
Sep 17 18:00:10.942577 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"3aafaf6e13fc9b96d481fb1c99648052879547305781d473d26cc5d92ad488c5"}
Sep 17 18:00:10.942706 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"3a6971d1d1efb9c9b4acd47f5d9946643eba2dee98b7b052a4e27a9b7e95a4d9"}
Sep 17 18:00:10.942706 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"35ab21e4a581b711e36ce894804c2177b4bccb8e325b6167868fd30a706adf04"}
Sep 17 18:00:10.942835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"272bff48620dcdafdfd650233514cf1ea288a9b6873b74cdbbf4be6891e992a0"}
Sep 17 18:00:10.942835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"3b83bfe7d3dea9df4635d2018f46a7db7dc613ca1c761054e7aed38b7049bc4c"}
Sep 17 18:00:10.942835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:10.942809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0dffac96-c450-4e20-a67a-0839b65791c5","Type":"ContainerStarted","Data":"fa17f6091d37fa3e6d3ca584a49b43e130c4839b2b38bb26bd4dfbde50958382"}
Sep 17 18:00:11.530105 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.530047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.515084027 podStartE2EDuration="3.530025959s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.941932ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:10.986713946 +0000 UTC m=+222.311035382" watchObservedRunningTime="2026-09-17 18:00:11.530025959 +0000 UTC m=+222.854347385"
Sep 17 18:00:11.531230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.531209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"]
Sep 17 18:00:11.534856 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.534836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.539684 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.539665    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:00:11.539810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.539671    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:00:11.539810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.539730    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:00:11.540022 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.540008    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:00:11.540343 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.540324    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vg2tj\""
Sep 17 18:00:11.540899 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.540882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:00:11.554612 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.554589    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:00:11.572285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.572262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"]
Sep 17 18:00:11.628423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnqn\" (UniqueName: \"kubernetes.io/projected/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-kube-api-access-llnqn\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-serving-certs-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-federate-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-metrics-client-ca\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.628876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.628665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnqn\" (UniqueName: \"kubernetes.io/projected/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-kube-api-access-llnqn\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-serving-certs-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.729894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-federate-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.730151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.730151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.729983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-metrics-client-ca\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.730712 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.730684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-metrics-client-ca\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.730858 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.730689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-serving-certs-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.730927 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.730875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.732427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.732406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.732510 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.732428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-telemeter-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.732510 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.732413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-secret-telemeter-client\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.732581 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.732534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-federate-client-tls\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.745901 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.745883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnqn\" (UniqueName: \"kubernetes.io/projected/b926fbaf-c4a2-46ed-95a2-64605bd84ac4-kube-api-access-llnqn\") pod \"telemeter-client-6cfd497cc4-tlzf7\" (UID: \"b926fbaf-c4a2-46ed-95a2-64605bd84ac4\") " pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.844029 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.843940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"
Sep 17 18:00:11.905634 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.905609    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:11.905930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.905898    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="prometheus" containerID="cri-o://684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2" gracePeriod=600
Sep 17 18:00:11.905930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.905917    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="thanos-sidecar" containerID="cri-o://3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02" gracePeriod=600
Sep 17 18:00:11.906076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.905929    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy" containerID="cri-o://c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977" gracePeriod=600
Sep 17 18:00:11.906076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.905965    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-web" containerID="cri-o://086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a" gracePeriod=600
Sep 17 18:00:11.906076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.906064    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a" gracePeriod=600
Sep 17 18:00:11.906248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:11.906086    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="config-reloader" containerID="cri-o://8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad" gracePeriod=600
Sep 17 18:00:12.131289 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.128688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7"]
Sep 17 18:00:12.132141 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:00:12.132117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb926fbaf_c4a2_46ed_95a2_64605bd84ac4.slice/crio-429c98e0baf718e0ec9240ff4a6a059a4537704215a38afda626d12b3c40de90 WatchSource:0}: Error finding container 429c98e0baf718e0ec9240ff4a6a059a4537704215a38afda626d12b3c40de90: Status 404 returned error can't find the container with id 429c98e0baf718e0ec9240ff4a6a059a4537704215a38afda626d12b3c40de90
Sep 17 18:00:12.144337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.144317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:12.233675 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.233843 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.233843 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.233974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.233974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.234084 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.233979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvzfd\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234547    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.235131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.234726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file\") pod \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\" (UID: \"1bb5b275-3c8a-40b4-8034-01dc2ed76564\") "
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235022    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235041    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235057    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.235925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.236483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.236437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:12.236872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.236491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:12.237509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.237483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.237647 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.237626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.238134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.238108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.238134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.238122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:12.239186 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.239286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out" (OuterVolumeSpecName: "config-out") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:12.239355 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config" (OuterVolumeSpecName: "config") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.239432 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.239432 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.239601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.239435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd" (OuterVolumeSpecName: "kube-api-access-qvzfd") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "kube-api-access-qvzfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:12.240143 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.240128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.250132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.250109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config" (OuterVolumeSpecName: "web-config") pod "1bb5b275-3c8a-40b4-8034-01dc2ed76564" (UID: "1bb5b275-3c8a-40b4-8034-01dc2ed76564"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:12.335408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335379    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335405    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-db\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335419    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-metrics-client-certs\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335427    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335436    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335445    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzfd\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-kube-api-access-qvzfd\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335453    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-configmap-metrics-client-ca\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335463    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-grpc-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335473    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335481    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bb5b275-3c8a-40b4-8034-01dc2ed76564-tls-assets\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335490    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-web-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335497    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335505    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335513    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335522    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bb5b275-3c8a-40b4-8034-01dc2ed76564-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335531    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bb5b275-3c8a-40b4-8034-01dc2ed76564-config-out\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.335570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.335539    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bb5b275-3c8a-40b4-8034-01dc2ed76564-secret-kube-rbac-proxy\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.949919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.949877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7" event={"ID":"b926fbaf-c4a2-46ed-95a2-64605bd84ac4","Type":"ContainerStarted","Data":"429c98e0baf718e0ec9240ff4a6a059a4537704215a38afda626d12b3c40de90"}
Sep 17 18:00:12.954011 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.953987    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a" exitCode=0
Sep 17 18:00:12.954011 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954008    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977" exitCode=0
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954017    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a" exitCode=0
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954024    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02" exitCode=0
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954030    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad" exitCode=0
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954036    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2" exitCode=0
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"}
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954078    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"}
Sep 17 18:00:12.954176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"}
Sep 17 18:00:12.954609 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954200    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:12.954609 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"}
Sep 17 18:00:12.954609 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"}
Sep 17 18:00:12.954609 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"}
Sep 17 18:00:12.954609 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.954381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bb5b275-3c8a-40b4-8034-01dc2ed76564","Type":"ContainerDied","Data":"5b1d06e5884590e52832863530e7a667513de65e25b9dc2ef2f72ace08effecd"}
Sep 17 18:00:12.963542 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.963526    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:12.971440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.971421    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:12.978106 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.978089    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:12.984854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.984829    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:12.985893 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.985873    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:12.991418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.991364    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:12.993711 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:12.993684    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.000855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.000837    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.007647 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.007630    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.007958 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.007939    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.008027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.007967    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.008027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.007986    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.008250 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.008231    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.008296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008257    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.008296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008273    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.008499 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.008483    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.008561 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008507    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.008561 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008527    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.008798 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.008740    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.008798 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008778    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.008798 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.008796    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.009050 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.009035    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.009098 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.009098 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009066    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.009345 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.009327    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.009388 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009349    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.009388 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009363    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.009589 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:00:13.009573    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.009633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009593    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.009633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009606    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.009828 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009812    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.009884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.009828    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.010037 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010018    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.010083 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010040    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.010248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010230    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.010248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010250    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.010480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010457    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.010480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010472    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.010688 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010667    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.010688 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010684    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.010907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010882    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.010907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.010903    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.011162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011147    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.011219 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011162    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.011394 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011377    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.011444 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011397    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.011593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011575    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.011661 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011594    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.011870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011846    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.011870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.011866    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.012150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.012217 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012152    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.012448 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012421    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.012448 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012445    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.012739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012721    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.012739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012740    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.013002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.012984    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.013071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013011    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.013281 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013263    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.013349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013282    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.013545 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.013545 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013541    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.013831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013811    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.013917 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.013830    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.014113 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014090    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.014178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014116    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.014352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.014397 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014353    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.014618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014599    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.014618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014618    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.014865 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014847    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.014915 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.014866    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.015098 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015081    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.015098 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015098    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.015314 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015295    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.015314 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015312    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.015522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015507    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.015522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015520    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.015736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015721    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.015802 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015736    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.015929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015910    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.015971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.015929    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.016137 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016122    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.016179 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016137    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.016346 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.016416 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016347    2577 scope.go:117] "RemoveContainer" containerID="1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"
Sep 17 18:00:13.016550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016533    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a"} err="failed to get container status \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": rpc error: code = NotFound desc = could not find container \"1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a\": container with ID starting with 1785e6beeae477d77f568bf89fbadfe97248f4376a6908706aac5df7c0126f0a not found: ID does not exist"
Sep 17 18:00:13.016616 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016551    2577 scope.go:117] "RemoveContainer" containerID="c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"
Sep 17 18:00:13.016783 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016767    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977"} err="failed to get container status \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": rpc error: code = NotFound desc = could not find container \"c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977\": container with ID starting with c4112954695785b2fc2d476c5aeb05c25f2821e03c6f771f09040ed80f661977 not found: ID does not exist"
Sep 17 18:00:13.016837 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016783    2577 scope.go:117] "RemoveContainer" containerID="086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"
Sep 17 18:00:13.017015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.016996    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a"} err="failed to get container status \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": rpc error: code = NotFound desc = could not find container \"086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a\": container with ID starting with 086a159aff66e7068f2d4963351eb818c31d55a62d3e7a389f83ab6053a9907a not found: ID does not exist"
Sep 17 18:00:13.017088 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017016    2577 scope.go:117] "RemoveContainer" containerID="3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"
Sep 17 18:00:13.017241 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017221    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02"} err="failed to get container status \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": rpc error: code = NotFound desc = could not find container \"3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02\": container with ID starting with 3e613c0a082a6f32e028b7ce24684d0e80cc15758d39477ee5e6944a0a33da02 not found: ID does not exist"
Sep 17 18:00:13.017285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017242    2577 scope.go:117] "RemoveContainer" containerID="8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"
Sep 17 18:00:13.017453 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017436    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad"} err="failed to get container status \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": rpc error: code = NotFound desc = could not find container \"8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad\": container with ID starting with 8d1889684e700c4f200772d051623e4035aff2db81f7d4dd6a6582c6df0111ad not found: ID does not exist"
Sep 17 18:00:13.017522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017454    2577 scope.go:117] "RemoveContainer" containerID="684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"
Sep 17 18:00:13.017684 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017664    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2"} err="failed to get container status \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": rpc error: code = NotFound desc = could not find container \"684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2\": container with ID starting with 684fd7f1844173396138007183c2335c55be836fd865e9f855fdcd88751046c2 not found: ID does not exist"
Sep 17 18:00:13.017726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017684    2577 scope.go:117] "RemoveContainer" containerID="72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"
Sep 17 18:00:13.017943 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.017926    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f"} err="failed to get container status \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": rpc error: code = NotFound desc = could not find container \"72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f\": container with ID starting with 72c21690dbb6e7841e64345f5376cdcbc0290a2fc04da52287418fa303cacf0f not found: ID does not exist"
Sep 17 18:00:13.029517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.029488    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:13.029976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.029957    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="config-reloader"
Sep 17 18:00:13.029976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.029974    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="config-reloader"
Sep 17 18:00:13.029976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.029985    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.029994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="init-config-reloader"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="init-config-reloader"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030019    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-web"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-web"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030030    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030046    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="prometheus"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030057    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="prometheus"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030067    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="thanos-sidecar"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030075    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="thanos-sidecar"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030134    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-web"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030147    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="prometheus"
Sep 17 18:00:13.030151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030155    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="config-reloader"
Sep 17 18:00:13.030716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030164    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="thanos-sidecar"
Sep 17 18:00:13.030716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030172    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy"
Sep 17 18:00:13.030716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.030182    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:13.035896 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.035858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.040541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040233    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:00:13.040541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:00:13.040541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:00:13.040541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:00:13.040541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:00:13.040959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:00:13.040959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:00:13.040959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.040891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f96rp\""
Sep 17 18:00:13.041144 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.041065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:00:13.041313 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.041269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:00:13.041769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.041553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:00:13.041769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.041579    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:00:13.043357 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.043338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:00:13.043501 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.043481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8l2995kr12md1\""
Sep 17 18:00:13.045300 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.045276    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:00:13.048259 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.048240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:00:13.058799 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.058774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:13.142248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-config-out\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75tk\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-kube-api-access-q75tk\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-web-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.142710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.143160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.143160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.143160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.143160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.142850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-config-out\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75tk\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-kube-api-access-q75tk\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-web-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.243898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.243978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.244517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.245304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.244972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.246702 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.246674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.247676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-web-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.247743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.247994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76a372ad-f8e0-4000-9485-0ea78f875a54-config-out\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.248952 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.248908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-config\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.249085 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.249064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.249231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.249214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.249427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.249410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.249474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.249447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76a372ad-f8e0-4000-9485-0ea78f875a54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.250517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.250496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76a372ad-f8e0-4000-9485-0ea78f875a54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.251966 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.251946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75tk\" (UniqueName: \"kubernetes.io/projected/76a372ad-f8e0-4000-9485-0ea78f875a54-kube-api-access-q75tk\") pod \"prometheus-k8s-0\" (UID: \"76a372ad-f8e0-4000-9485-0ea78f875a54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.348686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.348653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:13.644131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.644086    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:13.853661 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:00:13.853628    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a372ad_f8e0_4000_9485_0ea78f875a54.slice/crio-18fac675147d35257fc84c37fd829576d3ef1eef0e01c5de8a77caf701967fde WatchSource:0}: Error finding container 18fac675147d35257fc84c37fd829576d3ef1eef0e01c5de8a77caf701967fde: Status 404 returned error can't find the container with id 18fac675147d35257fc84c37fd829576d3ef1eef0e01c5de8a77caf701967fde
Sep 17 18:00:13.962134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.962108    2577 generic.go:356] "Generic (PLEG): container finished" podID="76a372ad-f8e0-4000-9485-0ea78f875a54" containerID="e472af3721aaec74aed6e9537c28a7466222da59f7c74c9246f7d534f403aa4f" exitCode=0
Sep 17 18:00:13.962228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.962153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerDied","Data":"e472af3721aaec74aed6e9537c28a7466222da59f7c74c9246f7d534f403aa4f"}
Sep 17 18:00:13.962228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:13.962203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"18fac675147d35257fc84c37fd829576d3ef1eef0e01c5de8a77caf701967fde"}
Sep 17 18:00:14.942656 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.942620    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb5b275-3c8a-40b4-8034-01dc2ed76564" path="/var/lib/kubelet/pods/1bb5b275-3c8a-40b4-8034-01dc2ed76564/volumes"
Sep 17 18:00:14.968483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.968455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7" event={"ID":"b926fbaf-c4a2-46ed-95a2-64605bd84ac4","Type":"ContainerStarted","Data":"b976eb645f7b964e83499297519f09aa0f683a9a26a6ca9355f5ed69b93d4d7e"}
Sep 17 18:00:14.968644 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.968517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7" event={"ID":"b926fbaf-c4a2-46ed-95a2-64605bd84ac4","Type":"ContainerStarted","Data":"2c19611d17f7095c4818275ca7b1f46bec05747ae973765512acacfc61d1b149"}
Sep 17 18:00:14.968644 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.968541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7" event={"ID":"b926fbaf-c4a2-46ed-95a2-64605bd84ac4","Type":"ContainerStarted","Data":"64cea6623c44e15ca11e0855d3630101d4b97baa047879718d6ac2da25be2541"}
Sep 17 18:00:14.971630 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"10d386cdbb0326068e2dfc39239d7009322b7da91f9fe7a56d76dc443ea46b00"}
Sep 17 18:00:14.971708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"9a681f030ef6441357561f95a3b58e4fb949cebf9a7fabe3795dc6b933fabc23"}
Sep 17 18:00:14.971708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"968362fe7976ce95bc847a1fc602b6bc117911e4659a4541717acbdd707fe75c"}
Sep 17 18:00:14.971708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"00351f57e0e1536e4ca9b5a8bdb4c6a71ef629267e45dbebfffacaaf2c189169"}
Sep 17 18:00:14.971830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"1fc375ea896aa466ab1ad0e9618139a5f2351b3fdd1e311027fd5730d6c0bc9f"}
Sep 17 18:00:14.971830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.971776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76a372ad-f8e0-4000-9485-0ea78f875a54","Type":"ContainerStarted","Data":"46d11253a118885d7e0fa43c2fba84d359047f13374ef8ddf048121702d2abe4"}
Sep 17 18:00:14.997440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:14.997388    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cfd497cc4-tlzf7" podStartSLOduration=2.226347404 podStartE2EDuration="3.99737488s" totalImagesPullingTime="1.771027476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:14.996634566 +0000 UTC m=+226.320955996" watchObservedRunningTime="2026-09-17 18:00:14.99737488 +0000 UTC m=+226.321696302"
Sep 17 18:00:15.040742 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:15.040699    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.022508144 podStartE2EDuration="2.040687078s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.178934ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:15.038242567 +0000 UTC m=+226.362564024" watchObservedRunningTime="2026-09-17 18:00:15.040687078 +0000 UTC m=+226.365008533"
Sep 17 18:00:15.219491 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:15.219397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:18.349805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:18.349742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:26.585705 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:26.585673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:26.690795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:26.690729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:26.759922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:26.759898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:45.346448 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:45.346370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:45.435185 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:00:45.435148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:13.348834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:13.348799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:13.364299 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:13.364276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:14.223416 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:14.223389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:21.742495 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:21.742455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:21.821363 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:21.821320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.519872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:26.519840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.636289 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:26.636251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:29.903690 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:29.903657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:30.856876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:30.856852    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:01:33.715686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:33.715637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:33.784373 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:33.784341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:37.849560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:37.849513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:37.956729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:37.956671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:38.026700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:38.026656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.497975 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:46.497940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.596911 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:01:46.596879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:00.656596 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:02:00.656565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:33.871930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:02:33.871849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:35.100994 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:02:35.100957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:01.749841 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:01.749798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:01.952541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:01.952506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:07.962436 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:07.962397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:08.247392 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:08.247313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:08.573213 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:08.573135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:08.727055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:08.727020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:08.952368 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:08.952338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:09.115271 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:09.115237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:09.340647 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:09.340570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:09.495249 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:09.495200    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:09.651216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:09.651186    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:09.915025 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:09.914934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:11.616199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:11.616168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:43.113849 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:43.113769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:43.222716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:03:43.222686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.543523 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:00.543491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.642210 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:00.642180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:14.857263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:14.857229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:14.952678 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:14.952648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:25.581932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:25.581895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:25.691375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:25.691340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:25.817615 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:25.817585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:29.624067 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:29.624033    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:29.754163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:29.754125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:29.851264 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:29.851229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:30.081645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:30.081568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:30.904611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:30.904576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:31.103982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:04:31.103945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:04.589779 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:04.589719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:06.356175 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:06.356139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:48.757412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:48.757377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:48.955769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:48.955732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:49.172940 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:49.172912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:50.282495 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:50.282465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:50.409447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:05:50.409392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:06.455457 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.455427    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-t29mz"]
Sep 17 18:06:06.458866 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.458851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.461532 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.461513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:06:06.461646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.461562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:06:06.461646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.461569    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:06:06.462533 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.462513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:06:06.462533 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.462531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:06:06.467892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.467872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t29mz"]
Sep 17 18:06:06.553047 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.553011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdws\" (UniqueName: \"kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.553197 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.553055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.653569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.653540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdws\" (UniqueName: \"kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.653707 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.653579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.653905 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.653888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.662558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.662482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdws\" (UniqueName: \"kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws\") pod \"s3-init-t29mz\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") " pod="kserve/s3-init-t29mz"
Sep 17 18:06:06.767898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:06.767833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t29mz"
Sep 17 18:06:07.039511 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:07.039473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t29mz"]
Sep 17 18:06:07.041877 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:06:07.041848    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecec4b8b_3a67_4e58_8736_3476d86eb86a.slice/crio-8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879 WatchSource:0}: Error finding container 8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879: Status 404 returned error can't find the container with id 8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879
Sep 17 18:06:07.043696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:07.043681    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:06:07.387708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:07.387675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerStarted","Data":"8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879"}
Sep 17 18:06:11.407648 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:11.407612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerStarted","Data":"8d81ae0745540b7ebcb97e5d2b2cde4b8851c67b95fd11e4a75777586ed9a3a7"}
Sep 17 18:06:13.417584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:13.417551    2577 generic.go:356] "Generic (PLEG): container finished" podID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerID="8d81ae0745540b7ebcb97e5d2b2cde4b8851c67b95fd11e4a75777586ed9a3a7" exitCode=0
Sep 17 18:06:13.417981 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:13.417615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerDied","Data":"8d81ae0745540b7ebcb97e5d2b2cde4b8851c67b95fd11e4a75777586ed9a3a7"}
Sep 17 18:06:18.444645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:18.444559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerStarted","Data":"9d25ec1927b48a2e987f67d26e4bd08fd8983a2bd3b87890d3d3b8bfd194442e"}
Sep 17 18:06:18.463052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:18.463006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-t29mz" podStartSLOduration=2.742173199 podStartE2EDuration="12.462992259s" totalImagesPullingTime="8.047562664s" totalInitContainerRuntime="1.673256396s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:18.461883912 +0000 UTC m=+589.786205336" watchObservedRunningTime="2026-09-17 18:06:18.462992259 +0000 UTC m=+589.787313685"
Sep 17 18:06:53.751463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:53.751424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:53.850440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:06:53.850413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:23.712958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:23.712922    2577 generic.go:356] "Generic (PLEG): container finished" podID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerID="9d25ec1927b48a2e987f67d26e4bd08fd8983a2bd3b87890d3d3b8bfd194442e" exitCode=0
Sep 17 18:07:23.713381 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:23.712989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerDied","Data":"9d25ec1927b48a2e987f67d26e4bd08fd8983a2bd3b87890d3d3b8bfd194442e"}
Sep 17 18:07:24.835770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:24.835736    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t29mz"
Sep 17 18:07:24.932425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:24.932400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrdws\" (UniqueName: \"kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws\") pod \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") "
Sep 17 18:07:24.932580 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:24.932466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache\") pod \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\" (UID: \"ecec4b8b-3a67-4e58-8736-3476d86eb86a\") "
Sep 17 18:07:24.934541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:24.934512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws" (OuterVolumeSpecName: "kube-api-access-lrdws") pod "ecec4b8b-3a67-4e58-8736-3476d86eb86a" (UID: "ecec4b8b-3a67-4e58-8736-3476d86eb86a"). InnerVolumeSpecName "kube-api-access-lrdws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:24.955378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:24.955354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecec4b8b-3a67-4e58-8736-3476d86eb86a" (UID: "ecec4b8b-3a67-4e58-8736-3476d86eb86a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:25.033524 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:25.033448    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdws\" (UniqueName: \"kubernetes.io/projected/ecec4b8b-3a67-4e58-8736-3476d86eb86a-kube-api-access-lrdws\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:07:25.033524 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:25.033474    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecec4b8b-3a67-4e58-8736-3476d86eb86a-model-cache\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:07:25.722318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:25.722296    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t29mz"
Sep 17 18:07:25.722480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:25.722290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t29mz" event={"ID":"ecec4b8b-3a67-4e58-8736-3476d86eb86a","Type":"ContainerDied","Data":"8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879"}
Sep 17 18:07:25.722480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:25.722428    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff7136f95a1876d807f3c50b650717652807e60bafe4d1eb7ee401b58d53879"
Sep 17 18:07:26.488664 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:07:26.488634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:00.354771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.354716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-pxvpr"]
Sep 17 18:08:00.355242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.355046    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerName="download-hf-model"
Sep 17 18:08:00.355242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.355058    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerName="download-hf-model"
Sep 17 18:08:00.355242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.355075    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerName="s3-init"
Sep 17 18:08:00.355242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.355080    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerName="s3-init"
Sep 17 18:08:00.355242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.355134    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecec4b8b-3a67-4e58-8736-3476d86eb86a" containerName="s3-init"
Sep 17 18:08:00.357009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.356994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.361290 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.361267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:08:00.362070 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.362044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:08:00.362070 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.362054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:08:00.362214 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.362075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:08:00.368181 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.368165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 18:08:00.383481 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.383453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-pxvpr"]
Sep 17 18:08:00.426833 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.426802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.426994 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.426841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfs5\" (UniqueName: \"kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.528134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.528098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.528321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.528141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfs5\" (UniqueName: \"kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.528480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.528458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.540503 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.540480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfs5\" (UniqueName: \"kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5\") pod \"s3-tls-init-custom-pxvpr\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") " pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.665243 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.665204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:00.942512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:00.942487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-pxvpr"]
Sep 17 18:08:00.943024 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:08:00.943003    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf3ada0_0809_40db_a496_9337fe588b13.slice/crio-79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596 WatchSource:0}: Error finding container 79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596: Status 404 returned error can't find the container with id 79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596
Sep 17 18:08:01.873892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:01.873851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerStarted","Data":"ec3b81ee7a6a5f5513159c43c55bf9b4143d626c7ad3fc4eb04824773ef91c3b"}
Sep 17 18:08:01.874299 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:01.873904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerStarted","Data":"79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596"}
Sep 17 18:08:03.883640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:03.883606    2577 generic.go:356] "Generic (PLEG): container finished" podID="abf3ada0-0809-40db-a496-9337fe588b13" containerID="ec3b81ee7a6a5f5513159c43c55bf9b4143d626c7ad3fc4eb04824773ef91c3b" exitCode=0
Sep 17 18:08:03.884042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:03.883682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerDied","Data":"ec3b81ee7a6a5f5513159c43c55bf9b4143d626c7ad3fc4eb04824773ef91c3b"}
Sep 17 18:08:04.889465 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:04.889425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerStarted","Data":"3f851ab795878197dfb89a6b3522ced2525fbf0a8491095a89266d01d7cd304e"}
Sep 17 18:08:04.909235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:04.909182    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-pxvpr" podStartSLOduration=2.9869522059999998 podStartE2EDuration="4.90916883s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.922216624s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:04.907305246 +0000 UTC m=+696.231626670" watchObservedRunningTime="2026-09-17 18:08:04.90916883 +0000 UTC m=+696.233490253"
Sep 17 18:08:10.919056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:10.919022    2577 generic.go:356] "Generic (PLEG): container finished" podID="abf3ada0-0809-40db-a496-9337fe588b13" containerID="3f851ab795878197dfb89a6b3522ced2525fbf0a8491095a89266d01d7cd304e" exitCode=0
Sep 17 18:08:10.919550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:10.919073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerDied","Data":"3f851ab795878197dfb89a6b3522ced2525fbf0a8491095a89266d01d7cd304e"}
Sep 17 18:08:12.053958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.053931    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:12.135383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.135322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache\") pod \"abf3ada0-0809-40db-a496-9337fe588b13\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") "
Sep 17 18:08:12.135383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.135356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfs5\" (UniqueName: \"kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5\") pod \"abf3ada0-0809-40db-a496-9337fe588b13\" (UID: \"abf3ada0-0809-40db-a496-9337fe588b13\") "
Sep 17 18:08:12.137331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.137309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5" (OuterVolumeSpecName: "kube-api-access-rqfs5") pod "abf3ada0-0809-40db-a496-9337fe588b13" (UID: "abf3ada0-0809-40db-a496-9337fe588b13"). InnerVolumeSpecName "kube-api-access-rqfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:12.234374 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.234305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache" (OuterVolumeSpecName: "model-cache") pod "abf3ada0-0809-40db-a496-9337fe588b13" (UID: "abf3ada0-0809-40db-a496-9337fe588b13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:12.236213 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.236191    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf3ada0-0809-40db-a496-9337fe588b13-model-cache\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:12.236284 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.236218    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfs5\" (UniqueName: \"kubernetes.io/projected/abf3ada0-0809-40db-a496-9337fe588b13-kube-api-access-rqfs5\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:12.927964 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.927936    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pxvpr"
Sep 17 18:08:12.927964 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.927944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pxvpr" event={"ID":"abf3ada0-0809-40db-a496-9337fe588b13","Type":"ContainerDied","Data":"79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596"}
Sep 17 18:08:12.928166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:12.927979    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e5a708b0dd096584b666d1cab3746c7d81d1d273b2c02cb253e2c40c665596"
Sep 17 18:08:13.580841 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:13.580811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.343644 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:15.343616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:17.973371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973339    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-f9hjh"]
Sep 17 18:08:17.973730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973652    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf3ada0-0809-40db-a496-9337fe588b13" containerName="s3-tls-init-custom"
Sep 17 18:08:17.973730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973663    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf3ada0-0809-40db-a496-9337fe588b13" containerName="s3-tls-init-custom"
Sep 17 18:08:17.973730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973679    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf3ada0-0809-40db-a496-9337fe588b13" containerName="download-hf-model"
Sep 17 18:08:17.973730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973685    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf3ada0-0809-40db-a496-9337fe588b13" containerName="download-hf-model"
Sep 17 18:08:17.973882 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.973739    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf3ada0-0809-40db-a496-9337fe588b13" containerName="s3-tls-init-custom"
Sep 17 18:08:17.976637 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.976621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:17.979128 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.979105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:08:17.979355 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.979339    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:08:17.980234 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.980217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:08:17.980327 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.980265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:08:17.980327 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.980298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 18:08:17.984992 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:17.984972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f9hjh"]
Sep 17 18:08:18.076710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.076681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.076905 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.076734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwnm\" (UniqueName: \"kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.178045 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.178017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.178149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.178078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwnm\" (UniqueName: \"kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.178393 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.178376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.186379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.186358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwnm\" (UniqueName: \"kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm\") pod \"s3-tls-init-serving-f9hjh\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") " pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.286460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.286386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:18.555566 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.555540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f9hjh"]
Sep 17 18:08:18.557922 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:08:18.557889    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4fc806e_4daf_4df7_9cb5_d5127aa607c4.slice/crio-34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5 WatchSource:0}: Error finding container 34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5: Status 404 returned error can't find the container with id 34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5
Sep 17 18:08:18.955720 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.955686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerStarted","Data":"6c3b38625d9988174b3eaed262cd4a0c8bcc27dd5250cae3dd06a0d28e8d3dd6"}
Sep 17 18:08:18.955884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:18.955728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerStarted","Data":"34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5"}
Sep 17 18:08:20.966434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:20.966402    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerID="6c3b38625d9988174b3eaed262cd4a0c8bcc27dd5250cae3dd06a0d28e8d3dd6" exitCode=0
Sep 17 18:08:20.966841 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:20.966472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerDied","Data":"6c3b38625d9988174b3eaed262cd4a0c8bcc27dd5250cae3dd06a0d28e8d3dd6"}
Sep 17 18:08:21.972470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:21.972439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerStarted","Data":"c8c8fa8d8b5a2ec3d9b64ea05cfa2977e83a9a28d569d92cd4b2280ce6403d0c"}
Sep 17 18:08:21.996069 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:21.996026    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-f9hjh" podStartSLOduration=3.143697643 podStartE2EDuration="4.996011075s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.852313432s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:21.995600998 +0000 UTC m=+713.319922422" watchObservedRunningTime="2026-09-17 18:08:21.996011075 +0000 UTC m=+713.320332498"
Sep 17 18:08:25.990700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:25.990669    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerID="c8c8fa8d8b5a2ec3d9b64ea05cfa2977e83a9a28d569d92cd4b2280ce6403d0c" exitCode=0
Sep 17 18:08:25.991064 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:25.990743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerDied","Data":"c8c8fa8d8b5a2ec3d9b64ea05cfa2977e83a9a28d569d92cd4b2280ce6403d0c"}
Sep 17 18:08:27.125887 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.125862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:27.159221 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.159190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwnm\" (UniqueName: \"kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm\") pod \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") "
Sep 17 18:08:27.159383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.159274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache\") pod \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\" (UID: \"f4fc806e-4daf-4df7-9cb5-d5127aa607c4\") "
Sep 17 18:08:27.161331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.161293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm" (OuterVolumeSpecName: "kube-api-access-cfwnm") pod "f4fc806e-4daf-4df7-9cb5-d5127aa607c4" (UID: "f4fc806e-4daf-4df7-9cb5-d5127aa607c4"). InnerVolumeSpecName "kube-api-access-cfwnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:27.183629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.183601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4fc806e-4daf-4df7-9cb5-d5127aa607c4" (UID: "f4fc806e-4daf-4df7-9cb5-d5127aa607c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:27.260401 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.260327    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-model-cache\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:27.260401 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:27.260351    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwnm\" (UniqueName: \"kubernetes.io/projected/f4fc806e-4daf-4df7-9cb5-d5127aa607c4-kube-api-access-cfwnm\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.000072 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:28.000041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f9hjh" event={"ID":"f4fc806e-4daf-4df7-9cb5-d5127aa607c4","Type":"ContainerDied","Data":"34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5"}
Sep 17 18:08:28.000239 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:28.000081    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34efbecc1b7ea68775f42e313ded25a7fe284bc7686f3622add3bbb0a9813ef5"
Sep 17 18:08:28.000239 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:28.000095    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f9hjh"
Sep 17 18:08:28.407393 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:28.407356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:28.508130 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:28.508086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.038046 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.037976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.186657 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.186619    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:08:40.186993 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.186980    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerName="download-hf-model"
Sep 17 18:08:40.187044 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.186994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerName="download-hf-model"
Sep 17 18:08:40.187044 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.187013    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerName="s3-tls-init-serving"
Sep 17 18:08:40.187044 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.187018    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerName="s3-tls-init-serving"
Sep 17 18:08:40.187132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.187080    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fc806e-4daf-4df7-9cb5-d5127aa607c4" containerName="s3-tls-init-serving"
Sep 17 18:08:40.191817 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.191800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.198278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.198256    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:08:40.198497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.198477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 18:08:40.198552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.198497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:08:40.199240 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.199224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:08:40.199633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.199616    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:08:40.199812 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.199794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:08:40.199875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.199832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 18:08:40.207452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.207425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:08:40.274603 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.274573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.274842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.274626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.274842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.274697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.274842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.274741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xn5\" (UniqueName: \"kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.274842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.274781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.321021 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.320950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.356995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.356967    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:40.360552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.360538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.363040 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.363019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f44f56-predictor-serving-cert\""
Sep 17 18:08:40.363152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.363045    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f44f56-kube-rbac-proxy-sar-config\""
Sep 17 18:08:40.363152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.363143    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f44f56-dockercfg-8h6kx\""
Sep 17 18:08:40.363294 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.363279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f44f56\""
Sep 17 18:08:40.374008 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.373988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:40.375210 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xn5\" (UniqueName: \"kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375469 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375469 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:08:40.375418    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 18:08:40.375566 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:08:40.375491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls podName:bca147b6-65cd-438f-ac11-75198d787ad7 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:40.875472299 +0000 UTC m=+732.199793714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls") pod "isvc-sklearn-batcher-predictor-69f554dc84-lll6f" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 18:08:40.375628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.375951 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.375933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.376022 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.376000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.390854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.390833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xn5\" (UniqueName: \"kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.441472 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.441420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.476608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.476584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27d6w\" (UniqueName: \"kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.476788 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.476614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.476788 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.476636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.476788 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.476662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.476788 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.476743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.542512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.542485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.577685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.577617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27d6w\" (UniqueName: \"kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.577685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.577654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.577685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.577684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.577867 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.577720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.577867 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.577810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.578225 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.578188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.578440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.578422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.578515 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.578447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.580287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.580266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.587324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.587298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27d6w\" (UniqueName: \"kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w\") pod \"isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") " pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.670741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.670710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:40.880247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.880218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.882544 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.882520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-69f554dc84-lll6f\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:40.945557 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:40.945529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:40.946846 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:08:40.946818    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcecf84_99e7_4721_bb59_e1a09289cea1.slice/crio-2bdb015da4f345a5e430b44c18a46c9274b6b793ae3274334d2c69b71fb0e591 WatchSource:0}: Error finding container 2bdb015da4f345a5e430b44c18a46c9274b6b793ae3274334d2c69b71fb0e591: Status 404 returned error can't find the container with id 2bdb015da4f345a5e430b44c18a46c9274b6b793ae3274334d2c69b71fb0e591
Sep 17 18:08:41.058611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:41.058577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerStarted","Data":"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"}
Sep 17 18:08:41.059035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:41.058638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerStarted","Data":"2bdb015da4f345a5e430b44c18a46c9274b6b793ae3274334d2c69b71fb0e591"}
Sep 17 18:08:41.101679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:41.101648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:08:41.387934 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:41.387817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:08:41.390144 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:08:41.390123    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca147b6_65cd_438f_ac11_75198d787ad7.slice/crio-ff730c81551c55ca809fce3b6054489b46d542d4b13a4f7dd6d91531750429cc WatchSource:0}: Error finding container ff730c81551c55ca809fce3b6054489b46d542d4b13a4f7dd6d91531750429cc: Status 404 returned error can't find the container with id ff730c81551c55ca809fce3b6054489b46d542d4b13a4f7dd6d91531750429cc
Sep 17 18:08:42.064670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:42.064638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerStarted","Data":"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"}
Sep 17 18:08:42.065061 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:42.064707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerStarted","Data":"ff730c81551c55ca809fce3b6054489b46d542d4b13a4f7dd6d91531750429cc"}
Sep 17 18:08:44.073454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:44.073427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/0.log"
Sep 17 18:08:44.073830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:44.073461    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerID="fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1" exitCode=1
Sep 17 18:08:44.073830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:44.073525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerDied","Data":"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"}
Sep 17 18:08:44.970907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:44.970870    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:44.985631 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:44.985565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:45.079868 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:45.079840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/0.log"
Sep 17 18:08:45.080324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:45.079908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerStarted","Data":"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"}
Sep 17 18:08:45.080324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:45.080022    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer" containerID="cri-o://5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4" gracePeriod=30
Sep 17 18:08:46.085567 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:46.085538    2577 generic.go:356] "Generic (PLEG): container finished" podID="bca147b6-65cd-438f-ac11-75198d787ad7" containerID="ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098" exitCode=0
Sep 17 18:08:46.086060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:46.085579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerDied","Data":"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"}
Sep 17 18:08:47.578253 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.578226    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/1.log"
Sep 17 18:08:47.578849 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.578771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/0.log"
Sep 17 18:08:47.581014 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.580995    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:47.746520 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.746482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location\") pod \"4fcecf84-99e7-4721-bb59-e1a09289cea1\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") "
Sep 17 18:08:47.746730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.746538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert\") pod \"4fcecf84-99e7-4721-bb59-e1a09289cea1\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") "
Sep 17 18:08:47.746730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.746571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls\") pod \"4fcecf84-99e7-4721-bb59-e1a09289cea1\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") "
Sep 17 18:08:47.746730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.746610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config\") pod \"4fcecf84-99e7-4721-bb59-e1a09289cea1\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") "
Sep 17 18:08:47.746730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.746651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27d6w\" (UniqueName: \"kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w\") pod \"4fcecf84-99e7-4721-bb59-e1a09289cea1\" (UID: \"4fcecf84-99e7-4721-bb59-e1a09289cea1\") "
Sep 17 18:08:47.747493 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.747439    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fcecf84-99e7-4721-bb59-e1a09289cea1" (UID: "4fcecf84-99e7-4721-bb59-e1a09289cea1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:47.747784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.747738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fcecf84-99e7-4721-bb59-e1a09289cea1" (UID: "4fcecf84-99e7-4721-bb59-e1a09289cea1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:47.748618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.748198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fcecf84-99e7-4721-bb59-e1a09289cea1" (UID: "4fcecf84-99e7-4721-bb59-e1a09289cea1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:47.751907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.751880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fcecf84-99e7-4721-bb59-e1a09289cea1" (UID: "4fcecf84-99e7-4721-bb59-e1a09289cea1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:47.753958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.753904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w" (OuterVolumeSpecName: "kube-api-access-27d6w") pod "4fcecf84-99e7-4721-bb59-e1a09289cea1" (UID: "4fcecf84-99e7-4721-bb59-e1a09289cea1"). InnerVolumeSpecName "kube-api-access-27d6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:47.847488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.847403    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fcecf84-99e7-4721-bb59-e1a09289cea1-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:47.847488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.847447    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:47.847488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.847461    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fcecf84-99e7-4721-bb59-e1a09289cea1-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:47.847488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.847474    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:47.847488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:47.847488    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27d6w\" (UniqueName: \"kubernetes.io/projected/4fcecf84-99e7-4721-bb59-e1a09289cea1-kube-api-access-27d6w\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:08:48.102901 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.102727    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/1.log"
Sep 17 18:08:48.103270 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw_4fcecf84-99e7-4721-bb59-e1a09289cea1/storage-initializer/0.log"
Sep 17 18:08:48.103393 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103292    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerID="5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4" exitCode=1
Sep 17 18:08:48.103455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerDied","Data":"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"}
Sep 17 18:08:48.103508 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw" event={"ID":"4fcecf84-99e7-4721-bb59-e1a09289cea1","Type":"ContainerDied","Data":"2bdb015da4f345a5e430b44c18a46c9274b6b793ae3274334d2c69b71fb0e591"}
Sep 17 18:08:48.103564 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103510    2577 scope.go:117] "RemoveContainer" containerID="5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"
Sep 17 18:08:48.103635 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.103614    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"
Sep 17 18:08:48.117053 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.117033    2577 scope.go:117] "RemoveContainer" containerID="fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"
Sep 17 18:08:48.130568 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.130539    2577 scope.go:117] "RemoveContainer" containerID="5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"
Sep 17 18:08:48.130904 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:08:48.130873    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4\": container with ID starting with 5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4 not found: ID does not exist" containerID="5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"
Sep 17 18:08:48.131002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.130916    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4"} err="failed to get container status \"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4\": rpc error: code = NotFound desc = could not find container \"5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4\": container with ID starting with 5221fb9611f5fdd866f5528e40c6b35f8d3101e5e239a897976db29a7eeab9b4 not found: ID does not exist"
Sep 17 18:08:48.131002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.130934    2577 scope.go:117] "RemoveContainer" containerID="fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"
Sep 17 18:08:48.131332 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:08:48.131307    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1\": container with ID starting with fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1 not found: ID does not exist" containerID="fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"
Sep 17 18:08:48.131418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.131342    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1"} err="failed to get container status \"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1\": rpc error: code = NotFound desc = could not find container \"fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1\": container with ID starting with fd6a76854e6561a183bd51fc0f329933e46b264b530cef30f4473b37886ee8e1 not found: ID does not exist"
Sep 17 18:08:48.148989 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.148958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:48.152998 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.152955    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f44f56-predictor-789cf79bd6-rpngw"]
Sep 17 18:08:48.945015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:48.944632    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" path="/var/lib/kubelet/pods/4fcecf84-99e7-4721-bb59-e1a09289cea1/volumes"
Sep 17 18:08:50.123679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.123604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:50.154793 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.154696    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:08:50.155216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155197    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.155216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.155380 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.155380 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155350    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.155482 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155444    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.155482 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.155454    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcecf84-99e7-4721-bb59-e1a09289cea1" containerName="storage-initializer"
Sep 17 18:08:50.177868 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.177471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:08:50.177868 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.177601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.182852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.181837    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 18:08:50.182852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.181966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 18:08:50.275266 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.275454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9k8\" (UniqueName: \"kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.275454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.275454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.275640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.275640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.275578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:50.377051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.376791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.376867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9k8\" (UniqueName: \"kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.376923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.376953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.376984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.377659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.377924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.377819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.378282 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.378240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.380253 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.380231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.386697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.386670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9k8\" (UniqueName: \"kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-699wj\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:50.493769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:50.493711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:08:51.525488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:51.525235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:08:51.528122 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:08:51.528064    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bacba5_9452_4965_ad66_00a4a4a424cb.slice/crio-6e1cb4218de59ad97d8ae1944e27bd42540bd8eb72aa2e43659bc99c4882c25e WatchSource:0}: Error finding container 6e1cb4218de59ad97d8ae1944e27bd42540bd8eb72aa2e43659bc99c4882c25e: Status 404 returned error can't find the container with id 6e1cb4218de59ad97d8ae1944e27bd42540bd8eb72aa2e43659bc99c4882c25e
Sep 17 18:08:52.136228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:52.136187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerStarted","Data":"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"}
Sep 17 18:08:52.136436 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:08:52.136264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerStarted","Data":"6e1cb4218de59ad97d8ae1944e27bd42540bd8eb72aa2e43659bc99c4882c25e"}
Sep 17 18:09:00.183651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:00.183612    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerID="65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe" exitCode=0
Sep 17 18:09:00.184035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:00.183684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerDied","Data":"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"}
Sep 17 18:09:01.204646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:01.204590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerStarted","Data":"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"}
Sep 17 18:09:23.346589 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:23.346520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerStarted","Data":"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"}
Sep 17 18:09:23.348550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:23.348514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerStarted","Data":"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"}
Sep 17 18:09:23.348650 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:23.348575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerStarted","Data":"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"}
Sep 17 18:09:24.353539 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:24.353501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:09:24.353980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:24.353601    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:09:24.354818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:24.354783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:09:24.378155 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:24.378100    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podStartSLOduration=7.355434346 podStartE2EDuration="34.378080581s" totalImagesPullingTime="22.903950798s" totalInitContainerRuntime="4.118695437s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:24.375722293 +0000 UTC m=+775.700043717" watchObservedRunningTime="2026-09-17 18:09:24.378080581 +0000 UTC m=+775.702402006"
Sep 17 18:09:25.359148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:25.359105    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:09:26.365503 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:26.365469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerStarted","Data":"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"}
Sep 17 18:09:26.365965 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:26.365674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:09:26.365965 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:26.365812    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:09:26.366988 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:26.366966    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:26.393069 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:26.393018    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podStartSLOduration=12.868944441 podStartE2EDuration="46.393003267s" totalImagesPullingTime="29.745288644s" totalInitContainerRuntime="3.778770182s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:26.391331262 +0000 UTC m=+777.715652696" watchObservedRunningTime="2026-09-17 18:09:26.393003267 +0000 UTC m=+777.717324690"
Sep 17 18:09:27.370055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:27.369960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:09:27.370546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:27.370077    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:27.371064 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:27.371032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:27.375063 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:27.375045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:09:28.375732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:28.375694    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:28.376193 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:28.376059    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:29.381264 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:29.381224    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:29.381705 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:29.381539    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:30.364195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:30.364168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:09:30.364807 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:30.364774    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:09:39.381474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:39.381427    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:39.381888 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:39.381827    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:40.365095 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:40.365046    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:09:49.381645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:49.381607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:49.382134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:49.382099    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:50.364904 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:50.364865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:09:59.382097 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:59.382048    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:09:59.382533 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:09:59.382511    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:00.364734 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:00.364696    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:10:09.382138 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:09.382085    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:10:09.382620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:09.382596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:10.365401 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:10.365361    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:10:19.381660 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:19.381611    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:10:19.382148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:19.382113    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:20.365482 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:20.365442    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:10:29.382381 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:29.382350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:10:29.382784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:29.382670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:10:30.365184 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:30.365148    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:10:35.057132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.057098    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:10:35.057595 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.057482    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" containerID="cri-o://0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99" gracePeriod=30
Sep 17 18:10:35.057595 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.057506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" containerID="cri-o://5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a" gracePeriod=30
Sep 17 18:10:35.057595 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.057559    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" containerID="cri-o://5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf" gracePeriod=30
Sep 17 18:10:35.126610 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.126522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:35.201741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.201702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:35.206608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.206586    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:10:35.209489 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.209474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.212065 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.212044    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 18:10:35.212346 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.212328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 18:10:35.220978 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.220956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:10:35.266375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.266345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:35.315980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.315903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.315980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.315945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.316163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.315987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.316163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.316016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.316163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.316052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lkm\" (UniqueName: \"kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417261 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417529 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417529 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lkm\" (UniqueName: \"kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.417846 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.417827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.418045 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.418025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.418152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.418132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.419783 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.419763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.426335 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.426315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lkm\" (UniqueName: \"kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm\") pod \"isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.520165 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.520135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:35.688188 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.688157    2577 generic.go:356] "Generic (PLEG): container finished" podID="bca147b6-65cd-438f-ac11-75198d787ad7" containerID="5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf" exitCode=2
Sep 17 18:10:35.688364 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.688202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerDied","Data":"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"}
Sep 17 18:10:35.793724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:35.793701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:10:35.795905 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:10:35.795883    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6954b75_e35c_4285_afed_c17bf2b4793a.slice/crio-67c4b2443ba0b8712b4ce45cdc72a6d71530f463b11f7af283f3bbece8c7ebd0 WatchSource:0}: Error finding container 67c4b2443ba0b8712b4ce45cdc72a6d71530f463b11f7af283f3bbece8c7ebd0: Status 404 returned error can't find the container with id 67c4b2443ba0b8712b4ce45cdc72a6d71530f463b11f7af283f3bbece8c7ebd0
Sep 17 18:10:36.694308 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:36.694274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerStarted","Data":"f21c1deb51ca90d4738351bf1942c5c5e849b65fa3ef4999331e9847886d816a"}
Sep 17 18:10:36.694717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:36.694342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerStarted","Data":"67c4b2443ba0b8712b4ce45cdc72a6d71530f463b11f7af283f3bbece8c7ebd0"}
Sep 17 18:10:37.370452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:37.370412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:10:39.381901 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:39.381864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:10:39.382269 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:39.382131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:39.713570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:39.713470    2577 generic.go:356] "Generic (PLEG): container finished" podID="bca147b6-65cd-438f-ac11-75198d787ad7" containerID="0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99" exitCode=0
Sep 17 18:10:39.713570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:39.713533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerDied","Data":"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"}
Sep 17 18:10:40.365412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:40.365372    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:10:40.719700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:40.719605    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerID="f21c1deb51ca90d4738351bf1942c5c5e849b65fa3ef4999331e9847886d816a" exitCode=0
Sep 17 18:10:40.719700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:40.719682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerDied","Data":"f21c1deb51ca90d4738351bf1942c5c5e849b65fa3ef4999331e9847886d816a"}
Sep 17 18:10:41.727687 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.727651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerStarted","Data":"8804ef9f5cddb81fe679412f405c4e07b27fa42170651e16da740e7ac9a21806"}
Sep 17 18:10:41.728109 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.727715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerStarted","Data":"81fe05e826aa48f0b912a15d2c178c533ceeb4436f9dba23109ac485c8efc9e8"}
Sep 17 18:10:41.728109 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.727741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerStarted","Data":"5fbe9b0681c3c10ff97de71bc7b5f20cb47252fd760d7d9792b35e72e9340198"}
Sep 17 18:10:41.728109 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.728036    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:41.728229 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.728158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:41.730120 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.730090    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:41.753399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:41.753354    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podStartSLOduration=2.866723708 podStartE2EDuration="6.753342232s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.886618524s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:41.751173555 +0000 UTC m=+853.075494981" watchObservedRunningTime="2026-09-17 18:10:41.753342232 +0000 UTC m=+853.077663655"
Sep 17 18:10:42.370850 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:42.370809    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:10:42.733072 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:42.732975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:42.733517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:42.733476    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:42.733588 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:42.733531    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:10:43.738259 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:43.738219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:10:43.738735 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:43.738711    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:47.370862 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:47.370814    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:10:47.371374 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:47.370968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:10:47.937904 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:47.937869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:10:48.742505 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:48.742475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:10:48.743013 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:48.742982    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:10:48.743500 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:48.743473    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:49.382047 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:49.382003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:10:49.382368 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:49.382337    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:50.590913 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.590862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:10:50.591348 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.591143    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container" containerID="cri-o://f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1" gracePeriod=30
Sep 17 18:10:50.591348 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.591221    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kube-rbac-proxy" containerID="cri-o://de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651" gracePeriod=30
Sep 17 18:10:50.632437 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.632413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:50.785581 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.785549    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerID="de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651" exitCode=2
Sep 17 18:10:50.785770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:50.785622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerDied","Data":"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"}
Sep 17 18:10:52.370195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:52.370150    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:10:55.359519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.359483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 18:10:55.523560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.523536    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:10:55.594723 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.594692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn9k8\" (UniqueName: \"kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8\") pod \"d3bacba5-9452-4965-ad66-00a4a4a424cb\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") "
Sep 17 18:10:55.594921 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.594735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location\") pod \"d3bacba5-9452-4965-ad66-00a4a4a424cb\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") "
Sep 17 18:10:55.594921 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.594783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls\") pod \"d3bacba5-9452-4965-ad66-00a4a4a424cb\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") "
Sep 17 18:10:55.594921 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.594904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config\") pod \"d3bacba5-9452-4965-ad66-00a4a4a424cb\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") "
Sep 17 18:10:55.595101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.594956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert\") pod \"d3bacba5-9452-4965-ad66-00a4a4a424cb\" (UID: \"d3bacba5-9452-4965-ad66-00a4a4a424cb\") "
Sep 17 18:10:55.595101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.595067    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3bacba5-9452-4965-ad66-00a4a4a424cb" (UID: "d3bacba5-9452-4965-ad66-00a4a4a424cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:55.595233 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.595213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3bacba5-9452-4965-ad66-00a4a4a424cb" (UID: "d3bacba5-9452-4965-ad66-00a4a4a424cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:55.595348 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.595331    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:10:55.595348 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.595333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3bacba5-9452-4965-ad66-00a4a4a424cb" (UID: "d3bacba5-9452-4965-ad66-00a4a4a424cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:55.595429 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.595355    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bacba5-9452-4965-ad66-00a4a4a424cb-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:10:55.596929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.596908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3bacba5-9452-4965-ad66-00a4a4a424cb" (UID: "d3bacba5-9452-4965-ad66-00a4a4a424cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:55.597035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.596928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8" (OuterVolumeSpecName: "kube-api-access-rn9k8") pod "d3bacba5-9452-4965-ad66-00a4a4a424cb" (UID: "d3bacba5-9452-4965-ad66-00a4a4a424cb"). InnerVolumeSpecName "kube-api-access-rn9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:55.696727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.696661    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bacba5-9452-4965-ad66-00a4a4a424cb-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:10:55.696727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.696684    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn9k8\" (UniqueName: \"kubernetes.io/projected/d3bacba5-9452-4965-ad66-00a4a4a424cb-kube-api-access-rn9k8\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:10:55.696727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.696693    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3bacba5-9452-4965-ad66-00a4a4a424cb-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:10:55.767170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.767142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:55.789212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789186    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:10:55.789547 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789535    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kube-rbac-proxy"
Sep 17 18:10:55.789594 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789549    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kube-rbac-proxy"
Sep 17 18:10:55.789594 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789561    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="storage-initializer"
Sep 17 18:10:55.789594 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789567    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="storage-initializer"
Sep 17 18:10:55.789594 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789573    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container"
Sep 17 18:10:55.789594 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789578    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container"
Sep 17 18:10:55.789784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kserve-container"
Sep 17 18:10:55.789784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.789649    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerName="kube-rbac-proxy"
Sep 17 18:10:55.791989 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.791975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.794418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.794400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 18:10:55.794531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.794514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:10:55.802638 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.802620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:10:55.813104 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.813080    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3bacba5-9452-4965-ad66-00a4a4a424cb" containerID="f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1" exitCode=0
Sep 17 18:10:55.813193 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.813138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerDied","Data":"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"}
Sep 17 18:10:55.813193 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.813148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"
Sep 17 18:10:55.813193 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.813185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj" event={"ID":"d3bacba5-9452-4965-ad66-00a4a4a424cb","Type":"ContainerDied","Data":"6e1cb4218de59ad97d8ae1944e27bd42540bd8eb72aa2e43659bc99c4882c25e"}
Sep 17 18:10:55.813295 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.813213    2577 scope.go:117] "RemoveContainer" containerID="de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"
Sep 17 18:10:55.822107 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.822042    2577 scope.go:117] "RemoveContainer" containerID="f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"
Sep 17 18:10:55.830336 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.830320    2577 scope.go:117] "RemoveContainer" containerID="65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"
Sep 17 18:10:55.838248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.838229    2577 scope.go:117] "RemoveContainer" containerID="de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"
Sep 17 18:10:55.838512 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:10:55.838493    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651\": container with ID starting with de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651 not found: ID does not exist" containerID="de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"
Sep 17 18:10:55.838559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.838524    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651"} err="failed to get container status \"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651\": rpc error: code = NotFound desc = could not find container \"de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651\": container with ID starting with de35d0afcbe0d773f47597a9813b9e590362da5d6b3b6b16af8fc527c2772651 not found: ID does not exist"
Sep 17 18:10:55.838559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.838546    2577 scope.go:117] "RemoveContainer" containerID="f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"
Sep 17 18:10:55.838848 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:10:55.838830    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1\": container with ID starting with f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1 not found: ID does not exist" containerID="f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"
Sep 17 18:10:55.838914 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.838853    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1"} err="failed to get container status \"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1\": rpc error: code = NotFound desc = could not find container \"f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1\": container with ID starting with f092e43265eb7828ef36d5b277654aa7dd31e8241161025dc1b56396aee93cd1 not found: ID does not exist"
Sep 17 18:10:55.838914 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.838870    2577 scope.go:117] "RemoveContainer" containerID="65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"
Sep 17 18:10:55.839032 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.839016    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:10:55.839129 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:10:55.839113    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe\": container with ID starting with 65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe not found: ID does not exist" containerID="65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"
Sep 17 18:10:55.839170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.839133    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe"} err="failed to get container status \"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe\": rpc error: code = NotFound desc = could not find container \"65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe\": container with ID starting with 65e9b18c9521d94ec0b39159b20dff19d8921bea4944bac0c51f3ee9db4d2ffe not found: ID does not exist"
Sep 17 18:10:55.845831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.845813    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-699wj"]
Sep 17 18:10:55.898835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.898816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.898934 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.898854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zh5j\" (UniqueName: \"kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.898934 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.898923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.899008 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.898945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.899043 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.899007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.930012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.929980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:55.999771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.999687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.999886 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.999815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:55.999886 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.999863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.000007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.999892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.000007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:55.999945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zh5j\" (UniqueName: \"kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.000236 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.000218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.000332 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.000313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.000494 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.000475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.002290 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.002273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.008215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.008198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zh5j\" (UniqueName: \"kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j\") pod \"isvc-predictive-xgboost-predictor-695f978978-69tqn\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.102506 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.102467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:10:56.373399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.373372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:10:56.375384 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:10:56.375363    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a63b3e_8dcc_4f37_b975_2993574c9dd6.slice/crio-6bafb01e403004b0b83e9432e405db897d00f323c89db9f6e6d65110ad02a4de WatchSource:0}: Error finding container 6bafb01e403004b0b83e9432e405db897d00f323c89db9f6e6d65110ad02a4de: Status 404 returned error can't find the container with id 6bafb01e403004b0b83e9432e405db897d00f323c89db9f6e6d65110ad02a4de
Sep 17 18:10:56.820577 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.820489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerStarted","Data":"8c14e7a7f097a971ed6505bbc1bf2bc7d12634acc1fb8c8912d3be763d0655bc"}
Sep 17 18:10:56.820577 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.820559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerStarted","Data":"6bafb01e403004b0b83e9432e405db897d00f323c89db9f6e6d65110ad02a4de"}
Sep 17 18:10:56.942987 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:56.942948    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bacba5-9452-4965-ad66-00a4a4a424cb" path="/var/lib/kubelet/pods/d3bacba5-9452-4965-ad66-00a4a4a424cb/volumes"
Sep 17 18:10:57.370597 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:57.370552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:10:58.743629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:58.743592    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:10:58.744043 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:58.744005    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:59.381877 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:59.381834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:10:59.382072 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:59.381966    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:10:59.382218 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:59.382194    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:10:59.382328 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:10:59.382313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:11:00.840928 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:00.840901    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerID="8c14e7a7f097a971ed6505bbc1bf2bc7d12634acc1fb8c8912d3be763d0655bc" exitCode=0
Sep 17 18:11:00.841297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:00.840976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerDied","Data":"8c14e7a7f097a971ed6505bbc1bf2bc7d12634acc1fb8c8912d3be763d0655bc"}
Sep 17 18:11:01.848116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.848086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerStarted","Data":"f88788a8f600062e484d2200799fd97a2561fc1a5bdc88ef387c6ee137c1cc32"}
Sep 17 18:11:01.848500 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.848154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerStarted","Data":"7a9791093f1050ed9bf1356fca006b83118e5f242d14c167464ac5a711886a5e"}
Sep 17 18:11:01.848500 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.848415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:11:01.848593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.848578    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:11:01.849894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.849871    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:01.871587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:01.871542    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podStartSLOduration=3.278766289 podStartE2EDuration="6.871528257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592761968s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:01.869270284 +0000 UTC m=+873.193591708" watchObservedRunningTime="2026-09-17 18:11:01.871528257 +0000 UTC m=+873.195849679"
Sep 17 18:11:02.370320 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:02.370277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:11:02.852976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:02.852884    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:05.198982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.198958    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:11:05.387772 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.387726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location\") pod \"bca147b6-65cd-438f-ac11-75198d787ad7\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") "
Sep 17 18:11:05.387772 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.387779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2xn5\" (UniqueName: \"kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5\") pod \"bca147b6-65cd-438f-ac11-75198d787ad7\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") "
Sep 17 18:11:05.388060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.387822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config\") pod \"bca147b6-65cd-438f-ac11-75198d787ad7\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") "
Sep 17 18:11:05.388060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.387844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert\") pod \"bca147b6-65cd-438f-ac11-75198d787ad7\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") "
Sep 17 18:11:05.388060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.387863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") pod \"bca147b6-65cd-438f-ac11-75198d787ad7\" (UID: \"bca147b6-65cd-438f-ac11-75198d787ad7\") "
Sep 17 18:11:05.388222 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.388107    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bca147b6-65cd-438f-ac11-75198d787ad7" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:05.388222 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.388131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bca147b6-65cd-438f-ac11-75198d787ad7" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:05.388222 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.388149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bca147b6-65cd-438f-ac11-75198d787ad7" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:05.389954 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.389928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5" (OuterVolumeSpecName: "kube-api-access-d2xn5") pod "bca147b6-65cd-438f-ac11-75198d787ad7" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7"). InnerVolumeSpecName "kube-api-access-d2xn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:05.390076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.390023    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bca147b6-65cd-438f-ac11-75198d787ad7" (UID: "bca147b6-65cd-438f-ac11-75198d787ad7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:05.488881 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.488847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:11:05.488881 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.488877    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca147b6-65cd-438f-ac11-75198d787ad7-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:11:05.488881 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.488888    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca147b6-65cd-438f-ac11-75198d787ad7-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:11:05.489084 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.488896    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca147b6-65cd-438f-ac11-75198d787ad7-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:11:05.489084 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.488906    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xn5\" (UniqueName: \"kubernetes.io/projected/bca147b6-65cd-438f-ac11-75198d787ad7-kube-api-access-d2xn5\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:11:05.870062 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.870028    2577 generic.go:356] "Generic (PLEG): container finished" podID="bca147b6-65cd-438f-ac11-75198d787ad7" containerID="5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a" exitCode=0
Sep 17 18:11:05.870230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.870084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"
Sep 17 18:11:05.870230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.870113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerDied","Data":"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"}
Sep 17 18:11:05.870230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.870192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f" event={"ID":"bca147b6-65cd-438f-ac11-75198d787ad7","Type":"ContainerDied","Data":"ff730c81551c55ca809fce3b6054489b46d542d4b13a4f7dd6d91531750429cc"}
Sep 17 18:11:05.870338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.870238    2577 scope.go:117] "RemoveContainer" containerID="5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"
Sep 17 18:11:05.880419 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.880404    2577 scope.go:117] "RemoveContainer" containerID="5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"
Sep 17 18:11:05.893501 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.893481    2577 scope.go:117] "RemoveContainer" containerID="0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"
Sep 17 18:11:05.902576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.902556    2577 scope.go:117] "RemoveContainer" containerID="ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"
Sep 17 18:11:05.906403 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.906383    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:11:05.912175 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912159    2577 scope.go:117] "RemoveContainer" containerID="5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"
Sep 17 18:11:05.912259 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912171    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-69f554dc84-lll6f"]
Sep 17 18:11:05.912473 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:11:05.912443    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a\": container with ID starting with 5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a not found: ID does not exist" containerID="5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"
Sep 17 18:11:05.912554 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912486    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a"} err="failed to get container status \"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a\": rpc error: code = NotFound desc = could not find container \"5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a\": container with ID starting with 5e5a0640ec412a64594b073a144ebe6175263fe452951e143ab27924213bb01a not found: ID does not exist"
Sep 17 18:11:05.912554 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912517    2577 scope.go:117] "RemoveContainer" containerID="5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"
Sep 17 18:11:05.912848 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:11:05.912831    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf\": container with ID starting with 5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf not found: ID does not exist" containerID="5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"
Sep 17 18:11:05.912899 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912853    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf"} err="failed to get container status \"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf\": rpc error: code = NotFound desc = could not find container \"5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf\": container with ID starting with 5a238e095eea8e5fcd22f140191339c27c67071640ab156f911d5a1d6ab120bf not found: ID does not exist"
Sep 17 18:11:05.912899 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.912867    2577 scope.go:117] "RemoveContainer" containerID="0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"
Sep 17 18:11:05.913099 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:11:05.913083    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99\": container with ID starting with 0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99 not found: ID does not exist" containerID="0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"
Sep 17 18:11:05.913160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.913103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99"} err="failed to get container status \"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99\": rpc error: code = NotFound desc = could not find container \"0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99\": container with ID starting with 0a48bc5ad034fe9e3eb62ecd6242480b4d90ebbe46f1d807fa8e303ee7ec6a99 not found: ID does not exist"
Sep 17 18:11:05.913160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.913117    2577 scope.go:117] "RemoveContainer" containerID="ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"
Sep 17 18:11:05.913413 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:11:05.913398    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098\": container with ID starting with ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098 not found: ID does not exist" containerID="ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"
Sep 17 18:11:05.913457 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:05.913417    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098"} err="failed to get container status \"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098\": rpc error: code = NotFound desc = could not find container \"ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098\": container with ID starting with ce94b5f3ebe33271b584eee77a9ab6702e75c4bacaffc418926cd8628cb53098 not found: ID does not exist"
Sep 17 18:11:06.941487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:06.941458    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" path="/var/lib/kubelet/pods/bca147b6-65cd-438f-ac11-75198d787ad7/volumes"
Sep 17 18:11:07.857662 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:07.857632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:11:07.857971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:07.857944    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:08.743157 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:08.743114    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:11:08.743592 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:08.743568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:17.858654 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:17.858575    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:18.743072 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:18.743031    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:11:18.743546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:18.743524    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:26.617153 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:26.617113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:27.858645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:27.858611    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:28.743829 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:28.743788    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:11:28.744205 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:28.744178    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:37.858106 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:37.858069    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:38.743345 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:38.743299    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:11:38.743725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:38.743690    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:47.858029 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:47.857984    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:48.743148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:48.743100    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:11:48.743683 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:48.743659    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:51.047558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:51.047525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:57.859049 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:57.859006    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:11:58.744450 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:58.744420    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:11:58.744623 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:11:58.744515    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:07.858584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:07.858546    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:12:10.230038 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.230005    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:12:10.230517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.230344    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" containerID="cri-o://5fbe9b0681c3c10ff97de71bc7b5f20cb47252fd760d7d9792b35e72e9340198" gracePeriod=30
Sep 17 18:12:10.230517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.230413    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" containerID="cri-o://81fe05e826aa48f0b912a15d2c178c533ceeb4436f9dba23109ac485c8efc9e8" gracePeriod=30
Sep 17 18:12:10.230517 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.230416    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" containerID="cri-o://8804ef9f5cddb81fe679412f405c4e07b27fa42170651e16da740e7ac9a21806" gracePeriod=30
Sep 17 18:12:10.241006 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.240965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:10.325278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.325245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:10.842056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:10.842017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.185378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:11.185347    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerID="81fe05e826aa48f0b912a15d2c178c533ceeb4436f9dba23109ac485c8efc9e8" exitCode=2
Sep 17 18:12:11.185549 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:11.185390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerDied","Data":"81fe05e826aa48f0b912a15d2c178c533ceeb4436f9dba23109ac485c8efc9e8"}
Sep 17 18:12:13.739128 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:13.739084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:15.207306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:15.207265    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerID="5fbe9b0681c3c10ff97de71bc7b5f20cb47252fd760d7d9792b35e72e9340198" exitCode=0
Sep 17 18:12:15.207675 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:15.207333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerDied","Data":"5fbe9b0681c3c10ff97de71bc7b5f20cb47252fd760d7d9792b35e72e9340198"}
Sep 17 18:12:17.858265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:17.858228    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:12:17.937407 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:17.937365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:12:18.739175 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:18.739131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:18.743964 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:18.743937    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:12:18.744299 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:18.744277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:23.738907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:23.738866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:23.739284 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:23.739005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:27.937889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:27.937863    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:12:28.738695 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:28.738651    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:28.743960 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:28.743922    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:12:28.744249 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:28.744226    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:30.320384 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.320355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:30.370058 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370033    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:12:30.370413 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370400    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="storage-initializer"
Sep 17 18:12:30.370460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370415    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="storage-initializer"
Sep 17 18:12:30.370460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370430    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy"
Sep 17 18:12:30.370460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370436    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy"
Sep 17 18:12:30.370460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370449    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container"
Sep 17 18:12:30.370460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370455    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container"
Sep 17 18:12:30.370645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370462    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent"
Sep 17 18:12:30.370645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370467    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent"
Sep 17 18:12:30.370645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370525    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="agent"
Sep 17 18:12:30.370645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370535    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kserve-container"
Sep 17 18:12:30.370645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.370541    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca147b6-65cd-438f-ac11-75198d787ad7" containerName="kube-rbac-proxy"
Sep 17 18:12:30.373893 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.373878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.376697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.376673    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 18:12:30.376914 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.376897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 18:12:30.386562 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.386541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:12:30.538389 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.538352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.538558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.538418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6vz\" (UniqueName: \"kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.538558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.538465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.538558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.538498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.538558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.538527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639793 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.639767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639969 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.639819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6vz\" (UniqueName: \"kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639969 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.639842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639969 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.639869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639969 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.639897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.639969 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:12:30.639935    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 18:12:30.640234 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:12:30.640010    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls podName:2ba05a47-b2de-42fb-babb-4b630edda7ba nodeName:}" failed. No retries permitted until 2026-09-17 18:12:31.139990681 +0000 UTC m=+962.464312083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls") pod "isvc-logger-predictor-657bc54f6c-q6tlk" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 18:12:30.640309 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.640290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.640535 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.640506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.640593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.640558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.648571 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.648549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6vz\" (UniqueName: \"kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:30.867112 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:30.867073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:31.145110 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:31.145075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:31.147479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:31.147448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") pod \"isvc-logger-predictor-657bc54f6c-q6tlk\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:31.283977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:31.283948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:31.558574 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:31.558552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:12:31.560434 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:12:31.560407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba05a47_b2de_42fb_babb_4b630edda7ba.slice/crio-9a72f9fd46ceba4b1fd36cccf428ae080892c344f565a4823383af13125e2f6e WatchSource:0}: Error finding container 9a72f9fd46ceba4b1fd36cccf428ae080892c344f565a4823383af13125e2f6e: Status 404 returned error can't find the container with id 9a72f9fd46ceba4b1fd36cccf428ae080892c344f565a4823383af13125e2f6e
Sep 17 18:12:32.286108 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:32.286073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerStarted","Data":"0cd21301933ed686072413011d5e2eb9c082d5bb7384f6dc8129ede9d121042f"}
Sep 17 18:12:32.286313 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:32.286147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerStarted","Data":"9a72f9fd46ceba4b1fd36cccf428ae080892c344f565a4823383af13125e2f6e"}
Sep 17 18:12:33.739433 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:33.739392    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:35.303441 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:35.303410    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerID="0cd21301933ed686072413011d5e2eb9c082d5bb7384f6dc8129ede9d121042f" exitCode=0
Sep 17 18:12:35.303808 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:35.303462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerDied","Data":"0cd21301933ed686072413011d5e2eb9c082d5bb7384f6dc8129ede9d121042f"}
Sep 17 18:12:36.286059 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.286026    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:12:36.286341 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.286306    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" containerID="cri-o://7a9791093f1050ed9bf1356fca006b83118e5f242d14c167464ac5a711886a5e" gracePeriod=30
Sep 17 18:12:36.286486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.286348    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kube-rbac-proxy" containerID="cri-o://f88788a8f600062e484d2200799fd97a2561fc1a5bdc88ef387c6ee137c1cc32" gracePeriod=30
Sep 17 18:12:36.299653 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.299629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:36.312048 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.312026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerStarted","Data":"512193d16828cc92c2dfbc177eec8308002cd70d5d7a9be3a909698c97a65875"}
Sep 17 18:12:36.312342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.312081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerStarted","Data":"8aeedb28e7b59b0a48c0345043ebdbf4d28660180bd65b51cfe8e6087bd1b177"}
Sep 17 18:12:36.312342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.312107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerStarted","Data":"28fb3d072f633db9a3dbc566ba8e08f64e7ade4140a0167b35df124cb4107d84"}
Sep 17 18:12:36.312342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.312298    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:36.312490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.312348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:36.313623 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.313601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:12:36.330685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:36.330648    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podStartSLOduration=2.693436404 podStartE2EDuration="6.330635249s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.637198845s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:36.329883969 +0000 UTC m=+967.654205394" watchObservedRunningTime="2026-09-17 18:12:36.330635249 +0000 UTC m=+967.654956651"
Sep 17 18:12:37.318870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.318836    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerID="f88788a8f600062e484d2200799fd97a2561fc1a5bdc88ef387c6ee137c1cc32" exitCode=2
Sep 17 18:12:37.319272 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.318906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerDied","Data":"f88788a8f600062e484d2200799fd97a2561fc1a5bdc88ef387c6ee137c1cc32"}
Sep 17 18:12:37.319272 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.319242    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:37.319418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.319392    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:12:37.320364 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.320334    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:37.853195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.853142    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 18:12:37.937821 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:37.937784    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:12:38.324261 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.324218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:12:38.324656 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.324567    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:38.739426 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.739391    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:12:38.743811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.743783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 18:12:38.743915 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.743902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:38.744124 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.744100    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:38.744204 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:38.744165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:40.336571 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.336535    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerID="8804ef9f5cddb81fe679412f405c4e07b27fa42170651e16da740e7ac9a21806" exitCode=0
Sep 17 18:12:40.336942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.336605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerDied","Data":"8804ef9f5cddb81fe679412f405c4e07b27fa42170651e16da740e7ac9a21806"}
Sep 17 18:12:40.382287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.382262    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:40.525640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.525554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert\") pod \"b6954b75-e35c-4285-afed-c17bf2b4793a\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") "
Sep 17 18:12:40.525640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.525602    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4lkm\" (UniqueName: \"kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm\") pod \"b6954b75-e35c-4285-afed-c17bf2b4793a\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") "
Sep 17 18:12:40.525640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.525638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls\") pod \"b6954b75-e35c-4285-afed-c17bf2b4793a\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") "
Sep 17 18:12:40.525927 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.525686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config\") pod \"b6954b75-e35c-4285-afed-c17bf2b4793a\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") "
Sep 17 18:12:40.525927 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.525733    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location\") pod \"b6954b75-e35c-4285-afed-c17bf2b4793a\" (UID: \"b6954b75-e35c-4285-afed-c17bf2b4793a\") "
Sep 17 18:12:40.526130 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.526100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6954b75-e35c-4285-afed-c17bf2b4793a" (UID: "b6954b75-e35c-4285-afed-c17bf2b4793a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:40.526195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.526118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6954b75-e35c-4285-afed-c17bf2b4793a" (UID: "b6954b75-e35c-4285-afed-c17bf2b4793a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:40.526195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.526005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6954b75-e35c-4285-afed-c17bf2b4793a" (UID: "b6954b75-e35c-4285-afed-c17bf2b4793a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:40.527817 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.527792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm" (OuterVolumeSpecName: "kube-api-access-g4lkm") pod "b6954b75-e35c-4285-afed-c17bf2b4793a" (UID: "b6954b75-e35c-4285-afed-c17bf2b4793a"). InnerVolumeSpecName "kube-api-access-g4lkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:40.527919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.527822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6954b75-e35c-4285-afed-c17bf2b4793a" (UID: "b6954b75-e35c-4285-afed-c17bf2b4793a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:40.626948 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.626910    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6954b75-e35c-4285-afed-c17bf2b4793a-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:40.626948 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.626940    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:40.626948 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.626951    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6954b75-e35c-4285-afed-c17bf2b4793a-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:40.627206 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.626959    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6954b75-e35c-4285-afed-c17bf2b4793a-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:40.627206 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:40.626969    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lkm\" (UniqueName: \"kubernetes.io/projected/b6954b75-e35c-4285-afed-c17bf2b4793a-kube-api-access-g4lkm\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.343808 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.343769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm" event={"ID":"b6954b75-e35c-4285-afed-c17bf2b4793a","Type":"ContainerDied","Data":"67c4b2443ba0b8712b4ce45cdc72a6d71530f463b11f7af283f3bbece8c7ebd0"}
Sep 17 18:12:41.344257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.343789    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"
Sep 17 18:12:41.344257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.343862    2577 scope.go:117] "RemoveContainer" containerID="8804ef9f5cddb81fe679412f405c4e07b27fa42170651e16da740e7ac9a21806"
Sep 17 18:12:41.346540 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.346511    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerID="7a9791093f1050ed9bf1356fca006b83118e5f242d14c167464ac5a711886a5e" exitCode=0
Sep 17 18:12:41.346666 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.346619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerDied","Data":"7a9791093f1050ed9bf1356fca006b83118e5f242d14c167464ac5a711886a5e"}
Sep 17 18:12:41.354055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.354037    2577 scope.go:117] "RemoveContainer" containerID="81fe05e826aa48f0b912a15d2c178c533ceeb4436f9dba23109ac485c8efc9e8"
Sep 17 18:12:41.363323 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.363240    2577 scope.go:117] "RemoveContainer" containerID="5fbe9b0681c3c10ff97de71bc7b5f20cb47252fd760d7d9792b35e72e9340198"
Sep 17 18:12:41.364878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.364857    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:12:41.372582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.372550    2577 scope.go:117] "RemoveContainer" containerID="f21c1deb51ca90d4738351bf1942c5c5e849b65fa3ef4999331e9847886d816a"
Sep 17 18:12:41.374189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.374165    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-57486b694b-hdrqm"]
Sep 17 18:12:41.436183 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.436161    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:12:41.445003 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.444981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.480658 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.480631    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:12:41.480994 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.480982    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.480996    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481009    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481025    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="storage-initializer"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481031    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="storage-initializer"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481039    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481050    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kube-rbac-proxy"
Sep 17 18:12:41.481052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481055    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kube-rbac-proxy"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481065    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481070    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="storage-initializer"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="storage-initializer"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481135    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="agent"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481143    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kserve-container"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481148    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" containerName="kube-rbac-proxy"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481155    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kserve-container"
Sep 17 18:12:41.481315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.481162    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" containerName="kube-rbac-proxy"
Sep 17 18:12:41.484329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.484314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.486572 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.486554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:12:41.486679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.486643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 18:12:41.500608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.500588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:12:41.533073 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zh5j\" (UniqueName: \"kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j\") pod \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") "
Sep 17 18:12:41.533073 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls\") pod \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") "
Sep 17 18:12:41.533073 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location\") pod \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") "
Sep 17 18:12:41.533265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config\") pod \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") "
Sep 17 18:12:41.533265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert\") pod \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\" (UID: \"d2a63b3e-8dcc-4f37-b975-2993574c9dd6\") "
Sep 17 18:12:41.533513 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2a63b3e-8dcc-4f37-b975-2993574c9dd6" (UID: "d2a63b3e-8dcc-4f37-b975-2993574c9dd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:41.533591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2a63b3e-8dcc-4f37-b975-2993574c9dd6" (UID: "d2a63b3e-8dcc-4f37-b975-2993574c9dd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:41.533591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.533520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2a63b3e-8dcc-4f37-b975-2993574c9dd6" (UID: "d2a63b3e-8dcc-4f37-b975-2993574c9dd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:41.535224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.535203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j" (OuterVolumeSpecName: "kube-api-access-4zh5j") pod "d2a63b3e-8dcc-4f37-b975-2993574c9dd6" (UID: "d2a63b3e-8dcc-4f37-b975-2993574c9dd6"). InnerVolumeSpecName "kube-api-access-4zh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:41.535317 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.535271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2a63b3e-8dcc-4f37-b975-2993574c9dd6" (UID: "d2a63b3e-8dcc-4f37-b975-2993574c9dd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:41.536180 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.536155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.634512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.634651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.634651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4wk8\" (UniqueName: \"kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.634729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.634729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.634729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634713    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.634729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634723    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.634884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634737    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zh5j\" (UniqueName: \"kubernetes.io/projected/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kube-api-access-4zh5j\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.634884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634766    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.634884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.634775    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a63b3e-8dcc-4f37-b975-2993574c9dd6-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.735474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.735441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4wk8\" (UniqueName: \"kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.735474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.735474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.735671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.735491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.735671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.735639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.735826 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.735724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.736117 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.736094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.736288 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.736267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.736321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.736267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.738038 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.738021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.743083 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.743062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4wk8\" (UniqueName: \"kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:41.794281 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:41.794216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:42.070016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.068596    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:12:42.072884 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:12:42.072833    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff451812_7703_4911_bf86_cb7f2307ac63.slice/crio-68d9f4d16d05b71e59af0d72429e7eb6d014eba75910f1fda1864bad6984d881 WatchSource:0}: Error finding container 68d9f4d16d05b71e59af0d72429e7eb6d014eba75910f1fda1864bad6984d881: Status 404 returned error can't find the container with id 68d9f4d16d05b71e59af0d72429e7eb6d014eba75910f1fda1864bad6984d881
Sep 17 18:12:42.354452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.354355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerStarted","Data":"f6135b259364eaf460d97519c26cd42d2c0ecbb04c37900b2a4baace3902b87f"}
Sep 17 18:12:42.354452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.354423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerStarted","Data":"68d9f4d16d05b71e59af0d72429e7eb6d014eba75910f1fda1864bad6984d881"}
Sep 17 18:12:42.357225 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.357198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn" event={"ID":"d2a63b3e-8dcc-4f37-b975-2993574c9dd6","Type":"ContainerDied","Data":"6bafb01e403004b0b83e9432e405db897d00f323c89db9f6e6d65110ad02a4de"}
Sep 17 18:12:42.357341 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.357264    2577 scope.go:117] "RemoveContainer" containerID="f88788a8f600062e484d2200799fd97a2561fc1a5bdc88ef387c6ee137c1cc32"
Sep 17 18:12:42.357341 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.357268    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"
Sep 17 18:12:42.365409 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.365385    2577 scope.go:117] "RemoveContainer" containerID="7a9791093f1050ed9bf1356fca006b83118e5f242d14c167464ac5a711886a5e"
Sep 17 18:12:42.375101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.375081    2577 scope.go:117] "RemoveContainer" containerID="8c14e7a7f097a971ed6505bbc1bf2bc7d12634acc1fb8c8912d3be763d0655bc"
Sep 17 18:12:42.386339 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.386316    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:12:42.390042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.390020    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-69tqn"]
Sep 17 18:12:42.942207 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.942178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6954b75-e35c-4285-afed-c17bf2b4793a" path="/var/lib/kubelet/pods/b6954b75-e35c-4285-afed-c17bf2b4793a/volumes"
Sep 17 18:12:42.942711 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:42.942697    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a63b3e-8dcc-4f37-b975-2993574c9dd6" path="/var/lib/kubelet/pods/d2a63b3e-8dcc-4f37-b975-2993574c9dd6/volumes"
Sep 17 18:12:43.328849 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:43.328771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:12:43.329322 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:43.329293    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:12:43.329701 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:43.329679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:46.380305 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:46.380274    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff451812-7703-4911-bf86-cb7f2307ac63" containerID="f6135b259364eaf460d97519c26cd42d2c0ecbb04c37900b2a4baace3902b87f" exitCode=0
Sep 17 18:12:46.380666 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:46.380346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerDied","Data":"f6135b259364eaf460d97519c26cd42d2c0ecbb04c37900b2a4baace3902b87f"}
Sep 17 18:12:47.397394 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.397361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerStarted","Data":"9013c577405575545fa20000e36f40cf938598f2b43f46e12475da7d4f7a2485"}
Sep 17 18:12:47.397795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.397444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerStarted","Data":"9e22b8b23fc4936fbd0678c8638e1395e26383d8cf9e181b8c9c1c042a715f2e"}
Sep 17 18:12:47.397795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.397746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:47.397891 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.397805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:47.399066 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.399040    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:12:47.415680 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:47.415635    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podStartSLOduration=2.674905657 podStartE2EDuration="6.415623595s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.740717938s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:47.415007255 +0000 UTC m=+978.739328679" watchObservedRunningTime="2026-09-17 18:12:47.415623595 +0000 UTC m=+978.739945018"
Sep 17 18:12:48.402454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:48.402416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:12:53.329273 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:53.329228    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:12:53.329828 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:53.329726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:53.406900 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:53.406876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:12:53.407411 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:53.407385    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:12:57.083536 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:12:57.083500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:03.329931 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:03.329886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:13:03.330519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:03.330413    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:03.407283 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:03.407245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:13:13.329842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:13.329796    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:13:13.330260 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:13.330230    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:13.407831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:13.407790    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:13:23.329338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:23.329294    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:13:23.329813 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:23.329770    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:23.407603 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:23.407561    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:13:33.330194 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:33.330147    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:13:33.330627 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:33.330554    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:33.407278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:33.407247    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:13:43.329619 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:43.329571    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:13:43.330148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:43.329998    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:43.407297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:43.407262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:13:53.329936 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:53.329896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:13:53.330400 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:53.330384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:13:53.407410 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:13:53.407365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:14:03.407917 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:03.407878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:14:05.587336 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.587301    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:14:05.587763 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.587551    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" containerID="cri-o://28fb3d072f633db9a3dbc566ba8e08f64e7ade4140a0167b35df124cb4107d84" gracePeriod=30
Sep 17 18:14:05.587763 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.587582    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" containerID="cri-o://8aeedb28e7b59b0a48c0345043ebdbf4d28660180bd65b51cfe8e6087bd1b177" gracePeriod=30
Sep 17 18:14:05.587763 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.587591    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" containerID="cri-o://512193d16828cc92c2dfbc177eec8308002cd70d5d7a9be3a909698c97a65875" gracePeriod=30
Sep 17 18:14:05.607878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.607847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:05.745573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.745538    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:14:05.749176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.749156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.751548 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.751526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 18:14:05.751548 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.751539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:14:05.760052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.760031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:14:05.776038 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.776009    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerID="8aeedb28e7b59b0a48c0345043ebdbf4d28660180bd65b51cfe8e6087bd1b177" exitCode=2
Sep 17 18:14:05.776134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.776078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerDied","Data":"8aeedb28e7b59b0a48c0345043ebdbf4d28660180bd65b51cfe8e6087bd1b177"}
Sep 17 18:14:05.847535 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.847446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.847535 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.847506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.847745 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.847540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.847745 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.847575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t446k\" (UniqueName: \"kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.847745 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.847615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.948659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.948624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.948871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.948665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.948871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.948690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t446k\" (UniqueName: \"kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.948871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.948705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.948871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.948767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.949135 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.949112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.949321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.949302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.949405 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.949387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.951128 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.951110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:05.957614 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:05.957593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t446k\" (UniqueName: \"kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bt645\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:06.059792 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:06.059744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:14:06.330966 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:06.330906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:14:06.333210 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:14:06.333183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5eedaf8_9817_47bf_924e_6c1ba126254e.slice/crio-07025475dc10f8e1519980f6216245f8ddd96f90797651a67e8d3c9f4e29fb19 WatchSource:0}: Error finding container 07025475dc10f8e1519980f6216245f8ddd96f90797651a67e8d3c9f4e29fb19: Status 404 returned error can't find the container with id 07025475dc10f8e1519980f6216245f8ddd96f90797651a67e8d3c9f4e29fb19
Sep 17 18:14:06.783073 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:06.782989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerStarted","Data":"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"}
Sep 17 18:14:06.783073 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:06.783054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerStarted","Data":"07025475dc10f8e1519980f6216245f8ddd96f90797651a67e8d3c9f4e29fb19"}
Sep 17 18:14:06.913986 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:06.913952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:08.325113 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:08.325072    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:10.804820 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:10.804784    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerID="28fb3d072f633db9a3dbc566ba8e08f64e7ade4140a0167b35df124cb4107d84" exitCode=0
Sep 17 18:14:10.805268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:10.804857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerDied","Data":"28fb3d072f633db9a3dbc566ba8e08f64e7ade4140a0167b35df124cb4107d84"}
Sep 17 18:14:11.811187 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.811152    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerID="b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0" exitCode=0
Sep 17 18:14:11.811677 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.811203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerDied","Data":"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"}
Sep 17 18:14:11.812495 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.812482    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:14:11.989947 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.989918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:14:11.990198 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.990172    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" containerID="cri-o://9e22b8b23fc4936fbd0678c8638e1395e26383d8cf9e181b8c9c1c042a715f2e" gracePeriod=30
Sep 17 18:14:11.990304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:11.990211    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kube-rbac-proxy" containerID="cri-o://9013c577405575545fa20000e36f40cf938598f2b43f46e12475da7d4f7a2485" gracePeriod=30
Sep 17 18:14:12.000552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:12.000520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:12.820645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:12.820563    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff451812-7703-4911-bf86-cb7f2307ac63" containerID="9013c577405575545fa20000e36f40cf938598f2b43f46e12475da7d4f7a2485" exitCode=2
Sep 17 18:14:12.820645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:12.820638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerDied","Data":"9013c577405575545fa20000e36f40cf938598f2b43f46e12475da7d4f7a2485"}
Sep 17 18:14:13.324930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:13.324881    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:13.329343 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:13.329307    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:14:13.331235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:13.331199    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:13.403247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:13.403189    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 18:14:13.407549 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:13.407511    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:14:17.149920 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:17.149883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:17.211598 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:17.211563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:17.855685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:17.855598    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff451812-7703-4911-bf86-cb7f2307ac63" containerID="9e22b8b23fc4936fbd0678c8638e1395e26383d8cf9e181b8c9c1c042a715f2e" exitCode=0
Sep 17 18:14:17.855870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:17.855675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerDied","Data":"9e22b8b23fc4936fbd0678c8638e1395e26383d8cf9e181b8c9c1c042a715f2e"}
Sep 17 18:14:18.043492 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.043464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:14:18.160418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160388    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location\") pod \"ff451812-7703-4911-bf86-cb7f2307ac63\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") "
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls\") pod \"ff451812-7703-4911-bf86-cb7f2307ac63\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") "
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4wk8\" (UniqueName: \"kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8\") pod \"ff451812-7703-4911-bf86-cb7f2307ac63\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") "
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config\") pod \"ff451812-7703-4911-bf86-cb7f2307ac63\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") "
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert\") pod \"ff451812-7703-4911-bf86-cb7f2307ac63\" (UID: \"ff451812-7703-4911-bf86-cb7f2307ac63\") "
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff451812-7703-4911-bf86-cb7f2307ac63" (UID: "ff451812-7703-4911-bf86-cb7f2307ac63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:18.160924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.160924    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff451812-7703-4911-bf86-cb7f2307ac63-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:18.161205 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.161014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff451812-7703-4911-bf86-cb7f2307ac63" (UID: "ff451812-7703-4911-bf86-cb7f2307ac63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:18.161205 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.161054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff451812-7703-4911-bf86-cb7f2307ac63" (UID: "ff451812-7703-4911-bf86-cb7f2307ac63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:18.162586 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.162558    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff451812-7703-4911-bf86-cb7f2307ac63" (UID: "ff451812-7703-4911-bf86-cb7f2307ac63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:18.162647 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.162602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8" (OuterVolumeSpecName: "kube-api-access-z4wk8") pod "ff451812-7703-4911-bf86-cb7f2307ac63" (UID: "ff451812-7703-4911-bf86-cb7f2307ac63"). InnerVolumeSpecName "kube-api-access-z4wk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:18.261590 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.261552    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4wk8\" (UniqueName: \"kubernetes.io/projected/ff451812-7703-4911-bf86-cb7f2307ac63-kube-api-access-z4wk8\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:18.261590 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.261580    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:18.261590 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.261590    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff451812-7703-4911-bf86-cb7f2307ac63-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:18.261590 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.261600    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff451812-7703-4911-bf86-cb7f2307ac63-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:18.325181 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.325139    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:18.325367 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.325280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:14:18.863224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.863194    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"
Sep 17 18:14:18.863224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.863209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96" event={"ID":"ff451812-7703-4911-bf86-cb7f2307ac63","Type":"ContainerDied","Data":"68d9f4d16d05b71e59af0d72429e7eb6d014eba75910f1fda1864bad6984d881"}
Sep 17 18:14:18.863480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.863278    2577 scope.go:117] "RemoveContainer" containerID="9013c577405575545fa20000e36f40cf938598f2b43f46e12475da7d4f7a2485"
Sep 17 18:14:18.872709 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.872692    2577 scope.go:117] "RemoveContainer" containerID="9e22b8b23fc4936fbd0678c8638e1395e26383d8cf9e181b8c9c1c042a715f2e"
Sep 17 18:14:18.880060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.880045    2577 scope.go:117] "RemoveContainer" containerID="f6135b259364eaf460d97519c26cd42d2c0ecbb04c37900b2a4baace3902b87f"
Sep 17 18:14:18.886525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.886499    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:14:18.889202 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.889185    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qdf96"]
Sep 17 18:14:18.942690 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:18.942658    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" path="/var/lib/kubelet/pods/ff451812-7703-4911-bf86-cb7f2307ac63/volumes"
Sep 17 18:14:23.324531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:23.324489    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:23.329975 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:23.329934    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:14:23.330667 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:23.330638    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:28.324650 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:28.324603    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:33.324836 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:33.324794    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:14:33.330150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:33.330120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:14:33.330260 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:33.330243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:14:33.330643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:33.330615    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:33.330777 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:33.330713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:14:35.955769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:35.955716    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerID="512193d16828cc92c2dfbc177eec8308002cd70d5d7a9be3a909698c97a65875" exitCode=0
Sep 17 18:14:35.956132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:35.955774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerDied","Data":"512193d16828cc92c2dfbc177eec8308002cd70d5d7a9be3a909698c97a65875"}
Sep 17 18:14:36.252497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.252461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:14:36.424449 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw6vz\" (UniqueName: \"kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz\") pod \"2ba05a47-b2de-42fb-babb-4b630edda7ba\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") "
Sep 17 18:14:36.424620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config\") pod \"2ba05a47-b2de-42fb-babb-4b630edda7ba\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") "
Sep 17 18:14:36.424620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location\") pod \"2ba05a47-b2de-42fb-babb-4b630edda7ba\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") "
Sep 17 18:14:36.424620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") pod \"2ba05a47-b2de-42fb-babb-4b630edda7ba\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") "
Sep 17 18:14:36.424782 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert\") pod \"2ba05a47-b2de-42fb-babb-4b630edda7ba\" (UID: \"2ba05a47-b2de-42fb-babb-4b630edda7ba\") "
Sep 17 18:14:36.424830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ba05a47-b2de-42fb-babb-4b630edda7ba" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.424830 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.424812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ba05a47-b2de-42fb-babb-4b630edda7ba" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:36.425027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.425004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ba05a47-b2de-42fb-babb-4b630edda7ba" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:36.425101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.425072    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ba05a47-b2de-42fb-babb-4b630edda7ba-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.425101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.425088    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.425218 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.425102    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.426587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.426567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ba05a47-b2de-42fb-babb-4b630edda7ba" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:36.426685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.426611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz" (OuterVolumeSpecName: "kube-api-access-hw6vz") pod "2ba05a47-b2de-42fb-babb-4b630edda7ba" (UID: "2ba05a47-b2de-42fb-babb-4b630edda7ba"). InnerVolumeSpecName "kube-api-access-hw6vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:36.526062 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.525980    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw6vz\" (UniqueName: \"kubernetes.io/projected/2ba05a47-b2de-42fb-babb-4b630edda7ba-kube-api-access-hw6vz\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.526062 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.526023    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ba05a47-b2de-42fb-babb-4b630edda7ba-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.962258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.962231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk" event={"ID":"2ba05a47-b2de-42fb-babb-4b630edda7ba","Type":"ContainerDied","Data":"9a72f9fd46ceba4b1fd36cccf428ae080892c344f565a4823383af13125e2f6e"}
Sep 17 18:14:36.962589 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.962275    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"
Sep 17 18:14:36.962589 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.962302    2577 scope.go:117] "RemoveContainer" containerID="512193d16828cc92c2dfbc177eec8308002cd70d5d7a9be3a909698c97a65875"
Sep 17 18:14:36.970348 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.970332    2577 scope.go:117] "RemoveContainer" containerID="8aeedb28e7b59b0a48c0345043ebdbf4d28660180bd65b51cfe8e6087bd1b177"
Sep 17 18:14:36.977642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.977627    2577 scope.go:117] "RemoveContainer" containerID="28fb3d072f633db9a3dbc566ba8e08f64e7ade4140a0167b35df124cb4107d84"
Sep 17 18:14:36.982958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.982939    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:14:36.985166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.985144    2577 scope.go:117] "RemoveContainer" containerID="0cd21301933ed686072413011d5e2eb9c082d5bb7384f6dc8129ede9d121042f"
Sep 17 18:14:36.992801 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:36.992717    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-657bc54f6c-q6tlk"]
Sep 17 18:14:38.943267 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:14:38.943232    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" path="/var/lib/kubelet/pods/2ba05a47-b2de-42fb-babb-4b630edda7ba/volumes"
Sep 17 18:16:17.460497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:17.460469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerStarted","Data":"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"}
Sep 17 18:16:18.467883 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:18.467839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerStarted","Data":"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"}
Sep 17 18:16:18.468266 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:18.468034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:18.488674 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:18.488627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podStartSLOduration=3.093076582 podStartE2EDuration="2m13.488611217s" totalImagesPullingTime="2m5.511896584s" totalInitContainerRuntime="4.883638051s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:18.486514834 +0000 UTC m=+1189.810836251" watchObservedRunningTime="2026-09-17 18:16:18.488611217 +0000 UTC m=+1189.812932640"
Sep 17 18:16:19.473310 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:19.473280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:19.474565 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:19.474530    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:16:20.477835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:20.477799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:16:25.482709 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:25.482683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:25.483272 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:25.483246    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:16:27.396449 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:27.396418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:32.427161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:32.427118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:32.519574 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:32.519532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.483767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:35.483724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:45.919853 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:45.919822    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:16:45.920321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:45.920059    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container" containerID="cri-o://257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0" gracePeriod=30
Sep 17 18:16:45.920321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:45.920080    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kube-rbac-proxy" containerID="cri-o://907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839" gracePeriod=30
Sep 17 18:16:45.941062 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:45.941027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.599208 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:46.599178    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerID="907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839" exitCode=2
Sep 17 18:16:46.599389 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:46.599254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerDied","Data":"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"}
Sep 17 18:16:47.395305 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.395277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:47.468068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.467981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config\") pod \"c5eedaf8-9817-47bf-924e-6c1ba126254e\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") "
Sep 17 18:16:47.468068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location\") pod \"c5eedaf8-9817-47bf-924e-6c1ba126254e\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") "
Sep 17 18:16:47.468068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert\") pod \"c5eedaf8-9817-47bf-924e-6c1ba126254e\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") "
Sep 17 18:16:47.468353 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t446k\" (UniqueName: \"kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k\") pod \"c5eedaf8-9817-47bf-924e-6c1ba126254e\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") "
Sep 17 18:16:47.468353 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls\") pod \"c5eedaf8-9817-47bf-924e-6c1ba126254e\" (UID: \"c5eedaf8-9817-47bf-924e-6c1ba126254e\") "
Sep 17 18:16:47.468447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5eedaf8-9817-47bf-924e-6c1ba126254e" (UID: "c5eedaf8-9817-47bf-924e-6c1ba126254e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:47.468447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.468418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5eedaf8-9817-47bf-924e-6c1ba126254e" (UID: "c5eedaf8-9817-47bf-924e-6c1ba126254e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:47.470282 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.470257    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5eedaf8-9817-47bf-924e-6c1ba126254e" (UID: "c5eedaf8-9817-47bf-924e-6c1ba126254e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:47.470382 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.470362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k" (OuterVolumeSpecName: "kube-api-access-t446k") pod "c5eedaf8-9817-47bf-924e-6c1ba126254e" (UID: "c5eedaf8-9817-47bf-924e-6c1ba126254e"). InnerVolumeSpecName "kube-api-access-t446k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:47.472847 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.472827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5eedaf8-9817-47bf-924e-6c1ba126254e" (UID: "c5eedaf8-9817-47bf-924e-6c1ba126254e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:47.569672 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.569636    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t446k\" (UniqueName: \"kubernetes.io/projected/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-api-access-t446k\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:16:47.569672 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.569666    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eedaf8-9817-47bf-924e-6c1ba126254e-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:16:47.569672 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.569677    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:16:47.569926 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.569686    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eedaf8-9817-47bf-924e-6c1ba126254e-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:16:47.569926 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.569696    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eedaf8-9817-47bf-924e-6c1ba126254e-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:16:47.605538 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.605501    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerID="257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0" exitCode=0
Sep 17 18:16:47.605685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.605547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerDied","Data":"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"}
Sep 17 18:16:47.605685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.605564    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"
Sep 17 18:16:47.605685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.605629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645" event={"ID":"c5eedaf8-9817-47bf-924e-6c1ba126254e","Type":"ContainerDied","Data":"07025475dc10f8e1519980f6216245f8ddd96f90797651a67e8d3c9f4e29fb19"}
Sep 17 18:16:47.605685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.605656    2577 scope.go:117] "RemoveContainer" containerID="907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"
Sep 17 18:16:47.614452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.614434    2577 scope.go:117] "RemoveContainer" containerID="257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"
Sep 17 18:16:47.621787 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.621770    2577 scope.go:117] "RemoveContainer" containerID="b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"
Sep 17 18:16:47.626724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.626688    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:16:47.629390 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.629373    2577 scope.go:117] "RemoveContainer" containerID="907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"
Sep 17 18:16:47.629667 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:16:47.629649    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839\": container with ID starting with 907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839 not found: ID does not exist" containerID="907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"
Sep 17 18:16:47.629748 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.629677    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839"} err="failed to get container status \"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839\": rpc error: code = NotFound desc = could not find container \"907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839\": container with ID starting with 907d17d456409b9f479bec0d384666ba8df9cf70cf81215f19b1ac8f13169839 not found: ID does not exist"
Sep 17 18:16:47.629748 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.629702    2577 scope.go:117] "RemoveContainer" containerID="257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"
Sep 17 18:16:47.629983 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:16:47.629967    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0\": container with ID starting with 257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0 not found: ID does not exist" containerID="257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"
Sep 17 18:16:47.630052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.629991    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0"} err="failed to get container status \"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0\": rpc error: code = NotFound desc = could not find container \"257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0\": container with ID starting with 257cfdfd9775df2425d7d4c2fd9e04e74e0661c4e6b55cc908c45be3cb8ca2c0 not found: ID does not exist"
Sep 17 18:16:47.630052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.630012    2577 scope.go:117] "RemoveContainer" containerID="b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"
Sep 17 18:16:47.630242 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:16:47.630220    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0\": container with ID starting with b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0 not found: ID does not exist" containerID="b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"
Sep 17 18:16:47.630283 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.630250    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0"} err="failed to get container status \"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0\": rpc error: code = NotFound desc = could not find container \"b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0\": container with ID starting with b02eea7914d031c9a6094813b2864e1f0cd6fded8788ea4b18f4f9544bc26fe0 not found: ID does not exist"
Sep 17 18:16:47.630607 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:47.630591    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bt645"]
Sep 17 18:16:48.941596 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:48.941565    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" path="/var/lib/kubelet/pods/c5eedaf8-9817-47bf-924e-6c1ba126254e/volumes"
Sep 17 18:16:51.062916 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:51.062883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:52.183740 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:16:52.183706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:42.722492 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:17:42.722455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:42.779027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:17:42.778996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:48.563805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:17:48.563768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:09.429143 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:09.429112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:22.601282 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:22.601256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:27.696565 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.696533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:27.756664 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.756634    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:18:27.757006 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.756992    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757008    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757021    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="storage-initializer"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757029    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="storage-initializer"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757040    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757046    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757055    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="storage-initializer"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757061    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="storage-initializer"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757068    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="storage-initializer"
Sep 17 18:18:27.757071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757073    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="storage-initializer"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757081    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757093    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757098    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757108    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757124    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757130    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757143    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757199    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757206    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff451812-7703-4911-bf86-cb7f2307ac63" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757213    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757219    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kube-rbac-proxy"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757223    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="agent"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757228    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5eedaf8-9817-47bf-924e-6c1ba126254e" containerName="kserve-container"
Sep 17 18:18:27.757424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.757234    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba05a47-b2de-42fb-babb-4b630edda7ba" containerName="kserve-container"
Sep 17 18:18:27.759375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.759360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.761731 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.761709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:27.761905 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.761881    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:27.762690 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.762673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:27.762819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.762696    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:18:27.762819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.762706    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 18:18:27.762819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.762709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:18:27.762819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.762804    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:27.770712 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.770694    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:18:27.807661 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.807641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:27.808805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.808786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.808876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.808824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.808876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.808853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.808876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.808872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.808975 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.808931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910174 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910276 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910276 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.910598 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.911009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.911009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.910999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.912708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.912686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:27.918406 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:27.918384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:28.069377 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:28.069291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:28.337686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:28.337658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:18:28.339287 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:18:28.339263    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809915fe_8ba3_4cb3_befa_24dd83799ea9.slice/crio-1b7eb6da5504b1fffef13517c736e4b3c03a46f325a6e47d5db5f8293b8be6f1 WatchSource:0}: Error finding container 1b7eb6da5504b1fffef13517c736e4b3c03a46f325a6e47d5db5f8293b8be6f1: Status 404 returned error can't find the container with id 1b7eb6da5504b1fffef13517c736e4b3c03a46f325a6e47d5db5f8293b8be6f1
Sep 17 18:18:29.034427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:29.034387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerStarted","Data":"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"}
Sep 17 18:18:29.034842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:29.034458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerStarted","Data":"1b7eb6da5504b1fffef13517c736e4b3c03a46f325a6e47d5db5f8293b8be6f1"}
Sep 17 18:18:33.052399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:33.052367    2577 generic.go:356] "Generic (PLEG): container finished" podID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerID="2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31" exitCode=0
Sep 17 18:18:33.052782 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:33.052405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerDied","Data":"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"}
Sep 17 18:18:34.058889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:34.058850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerStarted","Data":"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"}
Sep 17 18:18:34.059270 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:34.058938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerStarted","Data":"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"}
Sep 17 18:18:34.059270 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:34.059152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:34.059373 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:34.059291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:18:34.084118 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:34.084076    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podStartSLOduration=3.311586981 podStartE2EDuration="7.084061121s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77247414s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:34.082222207 +0000 UTC m=+1325.406543630" watchObservedRunningTime="2026-09-17 18:18:34.084061121 +0000 UTC m=+1325.408382543"
Sep 17 18:18:40.069499 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:18:40.069472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:19:10.070809 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:10.070768    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:19:20.070887 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:20.070850    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:19:30.070717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:30.070674    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:19:40.070600 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:40.070550    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:19:50.470309 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:50.470277    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:19:57.845009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:57.844969    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:19:57.845423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:57.845265    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" containerID="cri-o://5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8" gracePeriod=30
Sep 17 18:19:57.845423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:57.845320    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kube-rbac-proxy" containerID="cri-o://cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8" gracePeriod=30
Sep 17 18:19:57.885444 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:57.885416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:58.431861 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:58.431828    2577 generic.go:356] "Generic (PLEG): container finished" podID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerID="cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8" exitCode=2
Sep 17 18:19:58.432027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:58.431890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerDied","Data":"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"}
Sep 17 18:19:59.937879 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:19:59.937844    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:20:00.064596 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:00.064553    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:20:02.977474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:02.977443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:03.178241 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.178215    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:20:03.208156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location\") pod \"809915fe-8ba3-4cb3-befa-24dd83799ea9\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") "
Sep 17 18:20:03.208352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208166    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config\") pod \"809915fe-8ba3-4cb3-befa-24dd83799ea9\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") "
Sep 17 18:20:03.208352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert\") pod \"809915fe-8ba3-4cb3-befa-24dd83799ea9\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") "
Sep 17 18:20:03.208352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls\") pod \"809915fe-8ba3-4cb3-befa-24dd83799ea9\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") "
Sep 17 18:20:03.208352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn\") pod \"809915fe-8ba3-4cb3-befa-24dd83799ea9\" (UID: \"809915fe-8ba3-4cb3-befa-24dd83799ea9\") "
Sep 17 18:20:03.208596 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "809915fe-8ba3-4cb3-befa-24dd83799ea9" (UID: "809915fe-8ba3-4cb3-befa-24dd83799ea9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:03.208661 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "809915fe-8ba3-4cb3-befa-24dd83799ea9" (UID: "809915fe-8ba3-4cb3-befa-24dd83799ea9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:03.208820 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.208798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "809915fe-8ba3-4cb3-befa-24dd83799ea9" (UID: "809915fe-8ba3-4cb3-befa-24dd83799ea9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:03.210629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.210601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn" (OuterVolumeSpecName: "kube-api-access-7bkxn") pod "809915fe-8ba3-4cb3-befa-24dd83799ea9" (UID: "809915fe-8ba3-4cb3-befa-24dd83799ea9"). InnerVolumeSpecName "kube-api-access-7bkxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:03.210776 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.210732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "809915fe-8ba3-4cb3-befa-24dd83799ea9" (UID: "809915fe-8ba3-4cb3-befa-24dd83799ea9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:03.309338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.309242    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:03.309338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.309273    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/809915fe-8ba3-4cb3-befa-24dd83799ea9-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:03.309338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.309284    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/809915fe-8ba3-4cb3-befa-24dd83799ea9-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:03.309338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.309292    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/809915fe-8ba3-4cb3-befa-24dd83799ea9-kube-api-access-7bkxn\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:03.309338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.309301    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/809915fe-8ba3-4cb3-befa-24dd83799ea9-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:03.335737 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.335703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:03.455468 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.455431    2577 generic.go:356] "Generic (PLEG): container finished" podID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerID="5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8" exitCode=0
Sep 17 18:20:03.455641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.455496    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"
Sep 17 18:20:03.455641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.455508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerDied","Data":"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"}
Sep 17 18:20:03.455641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.455571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72" event={"ID":"809915fe-8ba3-4cb3-befa-24dd83799ea9","Type":"ContainerDied","Data":"1b7eb6da5504b1fffef13517c736e4b3c03a46f325a6e47d5db5f8293b8be6f1"}
Sep 17 18:20:03.455641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.455599    2577 scope.go:117] "RemoveContainer" containerID="cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"
Sep 17 18:20:03.464708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.464685    2577 scope.go:117] "RemoveContainer" containerID="5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"
Sep 17 18:20:03.472731 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.472711    2577 scope.go:117] "RemoveContainer" containerID="2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"
Sep 17 18:20:03.477071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.477047    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:20:03.481416 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.481386    2577 scope.go:117] "RemoveContainer" containerID="cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"
Sep 17 18:20:03.481740 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:20:03.481723    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8\": container with ID starting with cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8 not found: ID does not exist" containerID="cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"
Sep 17 18:20:03.481837 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.481767    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8"} err="failed to get container status \"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8\": rpc error: code = NotFound desc = could not find container \"cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8\": container with ID starting with cc88525d3d16e76ef2c9058791e85d6a55481585ceb23ce1c83829f01a33ddd8 not found: ID does not exist"
Sep 17 18:20:03.481837 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.481786    2577 scope.go:117] "RemoveContainer" containerID="5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"
Sep 17 18:20:03.482106 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:20:03.482086    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8\": container with ID starting with 5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8 not found: ID does not exist" containerID="5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"
Sep 17 18:20:03.482159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.482110    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8"} err="failed to get container status \"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8\": rpc error: code = NotFound desc = could not find container \"5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8\": container with ID starting with 5d6e674f3a3480f668be26495afec843561c9e8124762b0039b3408b0445b8a8 not found: ID does not exist"
Sep 17 18:20:03.482159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.482126    2577 scope.go:117] "RemoveContainer" containerID="2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"
Sep 17 18:20:03.482387 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:20:03.482371    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31\": container with ID starting with 2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31 not found: ID does not exist" containerID="2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"
Sep 17 18:20:03.482437 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.482392    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31"} err="failed to get container status \"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31\": rpc error: code = NotFound desc = could not find container \"2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31\": container with ID starting with 2b9b5c017cca84e32754db2929e53f1178d4ee2c1c1d6f0c6590ea7e03be7e31 not found: ID does not exist"
Sep 17 18:20:03.482509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:03.482492    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4sh72"]
Sep 17 18:20:04.942090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:04.942061    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" path="/var/lib/kubelet/pods/809915fe-8ba3-4cb3-befa-24dd83799ea9/volumes"
Sep 17 18:20:04.998827 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:04.998798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:12.810413 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:12.810304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:20.761744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.761707    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-66bdc827"]
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762030    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="storage-initializer"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762042    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="storage-initializer"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kube-rbac-proxy"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762065    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kube-rbac-proxy"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762083    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762088    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762139    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kserve-container"
Sep 17 18:20:20.762178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.762150    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="809915fe-8ba3-4cb3-befa-24dd83799ea9" containerName="kube-rbac-proxy"
Sep 17 18:20:20.764931 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.764914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:20.767509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.767489    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:20:20.767509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.767497    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:20:20.768445 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.768432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:20:20.770664 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.770643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-66bdc827"]
Sep 17 18:20:20.850658 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.850621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlsr\" (UniqueName: \"kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr\") pod \"curl-metrics-66bdc827\" (UID: \"4b912a35-aff1-4676-afd2-89cf4aaeea30\") " pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:20.951239 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.951210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlsr\" (UniqueName: \"kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr\") pod \"curl-metrics-66bdc827\" (UID: \"4b912a35-aff1-4676-afd2-89cf4aaeea30\") " pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:20.959260 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:20.959239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlsr\" (UniqueName: \"kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr\") pod \"curl-metrics-66bdc827\" (UID: \"4b912a35-aff1-4676-afd2-89cf4aaeea30\") " pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:21.074429 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:21.074350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:21.338537 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:21.338498    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-66bdc827"]
Sep 17 18:20:21.340748 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:20:21.340721    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b912a35_aff1_4676_afd2_89cf4aaeea30.slice/crio-62e0c9c06b0a1f60a08e47580be468dedb291444c03bdb70bb5da5a658099771 WatchSource:0}: Error finding container 62e0c9c06b0a1f60a08e47580be468dedb291444c03bdb70bb5da5a658099771: Status 404 returned error can't find the container with id 62e0c9c06b0a1f60a08e47580be468dedb291444c03bdb70bb5da5a658099771
Sep 17 18:20:21.342633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:21.342616    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:20:21.538087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:21.538047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-66bdc827" event={"ID":"4b912a35-aff1-4676-afd2-89cf4aaeea30","Type":"ContainerStarted","Data":"62e0c9c06b0a1f60a08e47580be468dedb291444c03bdb70bb5da5a658099771"}
Sep 17 18:20:22.543183 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:22.543151    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b912a35-aff1-4676-afd2-89cf4aaeea30" containerID="5ba0a1ed780b31a58336cb19beda3fc001f06e0c3105fcc651f515ed93f30665" exitCode=0
Sep 17 18:20:22.543567 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:22.543213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-66bdc827" event={"ID":"4b912a35-aff1-4676-afd2-89cf4aaeea30","Type":"ContainerDied","Data":"5ba0a1ed780b31a58336cb19beda3fc001f06e0c3105fcc651f515ed93f30665"}
Sep 17 18:20:23.664601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.664578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:23.687595 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.687573    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-66bdc827_4b912a35-aff1-4676-afd2-89cf4aaeea30/curl-metrics-66bdc827/0.log"
Sep 17 18:20:23.711930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.711906    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-66bdc827"]
Sep 17 18:20:23.721655 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.721631    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-66bdc827"]
Sep 17 18:20:23.777963 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.777941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjlsr\" (UniqueName: \"kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr\") pod \"4b912a35-aff1-4676-afd2-89cf4aaeea30\" (UID: \"4b912a35-aff1-4676-afd2-89cf4aaeea30\") "
Sep 17 18:20:23.779944 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.779926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr" (OuterVolumeSpecName: "kube-api-access-cjlsr") pod "4b912a35-aff1-4676-afd2-89cf4aaeea30" (UID: "4b912a35-aff1-4676-afd2-89cf4aaeea30"). InnerVolumeSpecName "kube-api-access-cjlsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:23.879230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:23.879210    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlsr\" (UniqueName: \"kubernetes.io/projected/4b912a35-aff1-4676-afd2-89cf4aaeea30-kube-api-access-cjlsr\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:24.552560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:24.552537    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e0c9c06b0a1f60a08e47580be468dedb291444c03bdb70bb5da5a658099771"
Sep 17 18:20:24.552560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:24.552550    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:24.562457 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:20:24.562430    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-66bdc827\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" podUID="4b912a35-aff1-4676-afd2-89cf4aaeea30" pod="kserve/curl-metrics-66bdc827"
Sep 17 18:20:24.941818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:24.941788    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b912a35-aff1-4676-afd2-89cf4aaeea30" path="/var/lib/kubelet/pods/4b912a35-aff1-4676-afd2-89cf4aaeea30/volumes"
Sep 17 18:20:29.096631 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.096597    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-ad595e55"]
Sep 17 18:20:29.097106 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.096944    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b912a35-aff1-4676-afd2-89cf4aaeea30" containerName="curl-metrics-66bdc827"
Sep 17 18:20:29.097106 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.096957    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b912a35-aff1-4676-afd2-89cf4aaeea30" containerName="curl-metrics-66bdc827"
Sep 17 18:20:29.097106 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.097021    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b912a35-aff1-4676-afd2-89cf4aaeea30" containerName="curl-metrics-66bdc827"
Sep 17 18:20:29.100121 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.100106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:29.102628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.102607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:20:29.102811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.102662    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:20:29.102811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.102716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:20:29.103805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.103779    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ad595e55"]
Sep 17 18:20:29.220837 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.220809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lpl\" (UniqueName: \"kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl\") pod \"curl-metrics-ad595e55\" (UID: \"fa9b2fe8-f673-47b6-b3fc-be4285529165\") " pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:29.321684 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.321647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lpl\" (UniqueName: \"kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl\") pod \"curl-metrics-ad595e55\" (UID: \"fa9b2fe8-f673-47b6-b3fc-be4285529165\") " pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:29.330373 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.330347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lpl\" (UniqueName: \"kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl\") pod \"curl-metrics-ad595e55\" (UID: \"fa9b2fe8-f673-47b6-b3fc-be4285529165\") " pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:29.410321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.410292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:29.682443 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:29.682422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ad595e55"]
Sep 17 18:20:29.684695 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:20:29.684667    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa9b2fe8_f673_47b6_b3fc_be4285529165.slice/crio-4b87153c0b9d467bf680f8fbd94db34ebefff37aa1da8ca7cd2e3d64326f303b WatchSource:0}: Error finding container 4b87153c0b9d467bf680f8fbd94db34ebefff37aa1da8ca7cd2e3d64326f303b: Status 404 returned error can't find the container with id 4b87153c0b9d467bf680f8fbd94db34ebefff37aa1da8ca7cd2e3d64326f303b
Sep 17 18:20:30.582102 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:30.582069    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa9b2fe8-f673-47b6-b3fc-be4285529165" containerID="4c2c706673d16b88b6e71a0bfebd70b67bf733df92425c262dded712e542faa0" exitCode=0
Sep 17 18:20:30.582472 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:30.582146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ad595e55" event={"ID":"fa9b2fe8-f673-47b6-b3fc-be4285529165","Type":"ContainerDied","Data":"4c2c706673d16b88b6e71a0bfebd70b67bf733df92425c262dded712e542faa0"}
Sep 17 18:20:30.582472 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:30.582184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ad595e55" event={"ID":"fa9b2fe8-f673-47b6-b3fc-be4285529165","Type":"ContainerStarted","Data":"4b87153c0b9d467bf680f8fbd94db34ebefff37aa1da8ca7cd2e3d64326f303b"}
Sep 17 18:20:31.711220 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.711196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:31.733918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.733893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-ad595e55_fa9b2fe8-f673-47b6-b3fc-be4285529165/curl-metrics-ad595e55/0.log"
Sep 17 18:20:31.758526 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.758492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-ad595e55"]
Sep 17 18:20:31.761858 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.761839    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-ad595e55"]
Sep 17 18:20:31.843205 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.843147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64lpl\" (UniqueName: \"kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl\") pod \"fa9b2fe8-f673-47b6-b3fc-be4285529165\" (UID: \"fa9b2fe8-f673-47b6-b3fc-be4285529165\") "
Sep 17 18:20:31.845137 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.845113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl" (OuterVolumeSpecName: "kube-api-access-64lpl") pod "fa9b2fe8-f673-47b6-b3fc-be4285529165" (UID: "fa9b2fe8-f673-47b6-b3fc-be4285529165"). InnerVolumeSpecName "kube-api-access-64lpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:31.944286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:31.944258    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lpl\" (UniqueName: \"kubernetes.io/projected/fa9b2fe8-f673-47b6-b3fc-be4285529165-kube-api-access-64lpl\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:20:32.591966 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:32.591940    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b87153c0b9d467bf680f8fbd94db34ebefff37aa1da8ca7cd2e3d64326f303b"
Sep 17 18:20:32.592143 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:32.591972    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:32.601792 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:20:32.601748    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-ad595e55\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" podUID="fa9b2fe8-f673-47b6-b3fc-be4285529165" pod="kserve/curl-metrics-ad595e55"
Sep 17 18:20:32.941312 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:32.941279    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9b2fe8-f673-47b6-b3fc-be4285529165" path="/var/lib/kubelet/pods/fa9b2fe8-f673-47b6-b3fc-be4285529165/volumes"
Sep 17 18:20:37.086159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.086126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:37.123584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.123552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:20:37.123922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.123907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9b2fe8-f673-47b6-b3fc-be4285529165" containerName="curl-metrics-ad595e55"
Sep 17 18:20:37.123978 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.123924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9b2fe8-f673-47b6-b3fc-be4285529165" containerName="curl-metrics-ad595e55"
Sep 17 18:20:37.124015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.124002    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9b2fe8-f673-47b6-b3fc-be4285529165" containerName="curl-metrics-ad595e55"
Sep 17 18:20:37.127268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.127254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.130049 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130026    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:20:37.130176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:20:37.130176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:37.130725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 18:20:37.130864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:20:37.130864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 18:20:37.130864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.130806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:37.136113 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.136093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:20:37.279114 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.279084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:37.287323 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.287302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.287398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.287339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.287398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.287361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zm4\" (UniqueName: \"kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.287398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.287379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.287509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.287440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zm4\" (UniqueName: \"kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.388849 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.388828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.389048 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.389028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.389154 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.389134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.391055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.391034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.399221 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.399199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zm4\" (UniqueName: \"kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4\") pod \"isvc-sklearn-predictor-7789f6859b-vlnc7\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.437822 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.437793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:37.463085 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.463045    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:37.711967 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:37.711942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:20:37.713467 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:20:37.713441    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb763e037_7a24_4a47_bba5_b1c731365c64.slice/crio-d06b647d5f6c4e301b8c0de983b41a038949e85aa6f0fadf703a4423e82f16eb WatchSource:0}: Error finding container d06b647d5f6c4e301b8c0de983b41a038949e85aa6f0fadf703a4423e82f16eb: Status 404 returned error can't find the container with id d06b647d5f6c4e301b8c0de983b41a038949e85aa6f0fadf703a4423e82f16eb
Sep 17 18:20:38.620769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:38.620731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerStarted","Data":"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"}
Sep 17 18:20:38.621164 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:38.620805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerStarted","Data":"d06b647d5f6c4e301b8c0de983b41a038949e85aa6f0fadf703a4423e82f16eb"}
Sep 17 18:20:42.641408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:42.641376    2577 generic.go:356] "Generic (PLEG): container finished" podID="b763e037-7a24-4a47-bba5-b1c731365c64" containerID="23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1" exitCode=0
Sep 17 18:20:42.641800 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:42.641456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerDied","Data":"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"}
Sep 17 18:20:42.644372 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:42.644351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:42.781333 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:42.781309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:43.647912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.647878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerStarted","Data":"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"}
Sep 17 18:20:43.648291 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.647937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerStarted","Data":"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"}
Sep 17 18:20:43.648291 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.648191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:43.648363 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.648307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:43.649356 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.649333    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:20:43.669611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:43.669572    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podStartSLOduration=2.6044773770000003 podStartE2EDuration="6.669560366s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.065082989s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:43.667593803 +0000 UTC m=+1454.991915229" watchObservedRunningTime="2026-09-17 18:20:43.669560366 +0000 UTC m=+1454.993881788"
Sep 17 18:20:44.652729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:44.652689    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:20:49.657089 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:49.657064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:20:49.657569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:49.657542    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:20:59.658037 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:20:59.657997    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:09.658294 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:09.658246    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:19.657993 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:19.657950    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:29.658236 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:29.658197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:39.658320 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:39.658281    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:49.657821 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:49.657782    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:21:50.940291 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:21:50.940248    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:22:00.941423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:00.941392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:22:07.205408 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:07.205376    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:22:07.205848 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:07.205589    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" containerID="cri-o://8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97" gracePeriod=30
Sep 17 18:22:07.205848 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:07.205664    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kube-rbac-proxy" containerID="cri-o://eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299" gracePeriod=30
Sep 17 18:22:07.213149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:07.213127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:08.019982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:08.019952    2577 generic.go:356] "Generic (PLEG): container finished" podID="b763e037-7a24-4a47-bba5-b1c731365c64" containerID="eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299" exitCode=2
Sep 17 18:22:08.020162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:08.020011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerDied","Data":"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"}
Sep 17 18:22:09.653285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:09.653246    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 18:22:10.940548 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:10.940513    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:22:11.643608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.643578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:22:11.782559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls\") pod \"b763e037-7a24-4a47-bba5-b1c731365c64\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") "
Sep 17 18:22:11.782559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782515    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5zm4\" (UniqueName: \"kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4\") pod \"b763e037-7a24-4a47-bba5-b1c731365c64\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") "
Sep 17 18:22:11.782559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config\") pod \"b763e037-7a24-4a47-bba5-b1c731365c64\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") "
Sep 17 18:22:11.782888 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert\") pod \"b763e037-7a24-4a47-bba5-b1c731365c64\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") "
Sep 17 18:22:11.782888 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location\") pod \"b763e037-7a24-4a47-bba5-b1c731365c64\" (UID: \"b763e037-7a24-4a47-bba5-b1c731365c64\") "
Sep 17 18:22:11.783000 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b763e037-7a24-4a47-bba5-b1c731365c64" (UID: "b763e037-7a24-4a47-bba5-b1c731365c64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:11.783043 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b763e037-7a24-4a47-bba5-b1c731365c64" (UID: "b763e037-7a24-4a47-bba5-b1c731365c64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:11.783043 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.782983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b763e037-7a24-4a47-bba5-b1c731365c64" (UID: "b763e037-7a24-4a47-bba5-b1c731365c64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:11.784708 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.784688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b763e037-7a24-4a47-bba5-b1c731365c64" (UID: "b763e037-7a24-4a47-bba5-b1c731365c64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:11.784818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.784795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4" (OuterVolumeSpecName: "kube-api-access-j5zm4") pod "b763e037-7a24-4a47-bba5-b1c731365c64" (UID: "b763e037-7a24-4a47-bba5-b1c731365c64"). InnerVolumeSpecName "kube-api-access-j5zm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:11.884012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.883985    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:22:11.884012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.884008    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b763e037-7a24-4a47-bba5-b1c731365c64-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:22:11.884012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.884018    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b763e037-7a24-4a47-bba5-b1c731365c64-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:22:11.884204 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.884027    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b763e037-7a24-4a47-bba5-b1c731365c64-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:22:11.884204 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:11.884036    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zm4\" (UniqueName: \"kubernetes.io/projected/b763e037-7a24-4a47-bba5-b1c731365c64-kube-api-access-j5zm4\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:22:12.038651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.038569    2577 generic.go:356] "Generic (PLEG): container finished" podID="b763e037-7a24-4a47-bba5-b1c731365c64" containerID="8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97" exitCode=0
Sep 17 18:22:12.038651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.038632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerDied","Data":"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"}
Sep 17 18:22:12.039161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.038649    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"
Sep 17 18:22:12.039161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.038692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7" event={"ID":"b763e037-7a24-4a47-bba5-b1c731365c64","Type":"ContainerDied","Data":"d06b647d5f6c4e301b8c0de983b41a038949e85aa6f0fadf703a4423e82f16eb"}
Sep 17 18:22:12.039161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.038732    2577 scope.go:117] "RemoveContainer" containerID="eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"
Sep 17 18:22:12.047205 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.047187    2577 scope.go:117] "RemoveContainer" containerID="8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"
Sep 17 18:22:12.054452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.054436    2577 scope.go:117] "RemoveContainer" containerID="23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"
Sep 17 18:22:12.060419 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.060385    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:22:12.062253 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.062181    2577 scope.go:117] "RemoveContainer" containerID="eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"
Sep 17 18:22:12.062555 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:22:12.062536    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299\": container with ID starting with eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299 not found: ID does not exist" containerID="eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"
Sep 17 18:22:12.062634 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.062563    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299"} err="failed to get container status \"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299\": rpc error: code = NotFound desc = could not find container \"eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299\": container with ID starting with eca1a6d0a3bb7644d55018e468e7547d3702fccc95e43a54a547e317e117d299 not found: ID does not exist"
Sep 17 18:22:12.062634 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.062581    2577 scope.go:117] "RemoveContainer" containerID="8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"
Sep 17 18:22:12.062914 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:22:12.062890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97\": container with ID starting with 8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97 not found: ID does not exist" containerID="8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"
Sep 17 18:22:12.062995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.062923    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97"} err="failed to get container status \"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97\": rpc error: code = NotFound desc = could not find container \"8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97\": container with ID starting with 8036e966a94587aa021a53bf92ca970a5e2da087d6f16d5b1aab00d58b620c97 not found: ID does not exist"
Sep 17 18:22:12.062995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.062946    2577 scope.go:117] "RemoveContainer" containerID="23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"
Sep 17 18:22:12.063338 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:22:12.063318    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1\": container with ID starting with 23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1 not found: ID does not exist" containerID="23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"
Sep 17 18:22:12.063397 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.063342    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1"} err="failed to get container status \"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1\": rpc error: code = NotFound desc = could not find container \"23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1\": container with ID starting with 23c38d79356a0c04695c5172e2bf2fd0a3e4a41caf73add912629fb8a636b3f1 not found: ID does not exist"
Sep 17 18:22:12.064607 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.064586    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7789f6859b-vlnc7"]
Sep 17 18:22:12.376695 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.376665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:22:12.377007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.376994    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="storage-initializer"
Sep 17 18:22:12.377056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377008    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="storage-initializer"
Sep 17 18:22:12.377056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377015    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container"
Sep 17 18:22:12.377056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377021    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container"
Sep 17 18:22:12.377056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377036    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kube-rbac-proxy"
Sep 17 18:22:12.377056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377042    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kube-rbac-proxy"
Sep 17 18:22:12.377251 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377097    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kube-rbac-proxy"
Sep 17 18:22:12.377251 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.377104    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" containerName="kserve-container"
Sep 17 18:22:12.381703 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.381689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.384102 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:12.384229 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384117    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:22:12.384301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384254    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:12.384366 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:22:12.384554 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 18:22:12.384658 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:12.384658 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.384573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:12.391166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.391147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:22:12.488185 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.488155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.488349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.488207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.488349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.488228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpqm\" (UniqueName: \"kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.488349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.488251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.488349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.488267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpqm\" (UniqueName: \"kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.589995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.589972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.590199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.590181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.590266 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.590250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.591885 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.591865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.597358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.597337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpqm\" (UniqueName: \"kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.692500 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.692429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:22:12.941744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.941711    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b763e037-7a24-4a47-bba5-b1c731365c64" path="/var/lib/kubelet/pods/b763e037-7a24-4a47-bba5-b1c731365c64/volumes"
Sep 17 18:22:12.977746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:12.977727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:22:12.980143 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:22:12.980114    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca66151_f3fc_4be2_930e_946090928e89.slice/crio-7e76ef1b1ed959a4375ccc53ab7c50f1b9b796b2edba5f3d5ad3c507cfe49cab WatchSource:0}: Error finding container 7e76ef1b1ed959a4375ccc53ab7c50f1b9b796b2edba5f3d5ad3c507cfe49cab: Status 404 returned error can't find the container with id 7e76ef1b1ed959a4375ccc53ab7c50f1b9b796b2edba5f3d5ad3c507cfe49cab
Sep 17 18:22:13.010057 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:13.010034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:13.045712 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:13.045668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerStarted","Data":"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"}
Sep 17 18:22:13.046142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:13.045736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerStarted","Data":"7e76ef1b1ed959a4375ccc53ab7c50f1b9b796b2edba5f3d5ad3c507cfe49cab"}
Sep 17 18:22:13.067242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:13.067212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:17.064550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:17.064471    2577 generic.go:356] "Generic (PLEG): container finished" podID="aca66151-f3fc-4be2-930e-946090928e89" containerID="aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e" exitCode=0
Sep 17 18:22:17.064550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:17.064512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerDied","Data":"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"}
Sep 17 18:22:49.538016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:49.537980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:54.673477 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:54.673447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:55.195470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:22:55.195426    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:25.734346 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:25.734314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerStarted","Data":"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"}
Sep 17 18:24:25.734831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:25.734405    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:24:25.734831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:25.734443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerStarted","Data":"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"}
Sep 17 18:24:25.734831 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:25.734478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:24:25.757161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:25.757092    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" podStartSLOduration=1.629392245 podStartE2EDuration="2m13.757077495s" totalImagesPullingTime="2m8.433550325s" totalInitContainerRuntime="3.694134925s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:25.75443061 +0000 UTC m=+1677.078752046" watchObservedRunningTime="2026-09-17 18:24:25.757077495 +0000 UTC m=+1677.081398917"
Sep 17 18:24:31.744361 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:31.744330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:24:31.923206 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:31.923175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:41.084724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:41.084691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:42.800212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:42.800129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:51.181675 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:51.181643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:56.207343 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:56.207314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:56.583949 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:24:56.583869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:01.770926 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:01.770872    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:25:11.747736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:11.747706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:25:12.522076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.522041    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:25:12.522331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.522289    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kube-rbac-proxy" containerID="cri-o://390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413" gracePeriod=30
Sep 17 18:25:12.522331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.522280    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kserve-container" containerID="cri-o://e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26" gracePeriod=30
Sep 17 18:25:12.547417 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.547389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:12.941942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.941910    2577 generic.go:356] "Generic (PLEG): container finished" podID="aca66151-f3fc-4be2-930e-946090928e89" containerID="390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413" exitCode=2
Sep 17 18:25:12.942306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:12.941979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerDied","Data":"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"}
Sep 17 18:25:16.739834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:16.739794    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 18:25:17.689530 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.689505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:17.727671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.727639    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:17.734942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.734919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.737242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.737214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 18:25:17.737350 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.737227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:25:17.740674 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.740653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:17.767216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.767190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.767324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.767232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667df\" (UniqueName: \"kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.767324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.767256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.767324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.767298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.767433 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.767331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.867841 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.867808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.867972 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.867860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667df\" (UniqueName: \"kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.867972 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.867882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.867972 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.867910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.867972 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.867943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.868187 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:17.867943    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 18:25:17.868187 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:17.868050    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls podName:6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116 nodeName:}" failed. No retries permitted until 2026-09-17 18:25:18.368034771 +0000 UTC m=+1729.692356186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls") pod "isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 18:25:17.868301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.868266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.868747 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.868723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.868817 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.868772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.902377 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.902350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667df\" (UniqueName: \"kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:17.987090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:17.987019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:18.371925 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:18.371899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:18.374165 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:18.374142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-577d78f5bd-77cjw\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:18.646250 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:18.646154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:18.920742 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:18.920717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:18.922647 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:25:18.922616    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de1b1b0_cce2_49c7_b2e6_aedcdcbfc116.slice/crio-2850aeaf2a0bb802270682dec1bd2ea18bbca605cffb56f94877873135bbc62a WatchSource:0}: Error finding container 2850aeaf2a0bb802270682dec1bd2ea18bbca605cffb56f94877873135bbc62a: Status 404 returned error can't find the container with id 2850aeaf2a0bb802270682dec1bd2ea18bbca605cffb56f94877873135bbc62a
Sep 17 18:25:18.967447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:18.967417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerStarted","Data":"2850aeaf2a0bb802270682dec1bd2ea18bbca605cffb56f94877873135bbc62a"}
Sep 17 18:25:19.567877 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.567855    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:25:19.580091 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config\") pod \"aca66151-f3fc-4be2-930e-946090928e89\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") "
Sep 17 18:25:19.580184 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls\") pod \"aca66151-f3fc-4be2-930e-946090928e89\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") "
Sep 17 18:25:19.580184 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert\") pod \"aca66151-f3fc-4be2-930e-946090928e89\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") "
Sep 17 18:25:19.580184 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location\") pod \"aca66151-f3fc-4be2-930e-946090928e89\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") "
Sep 17 18:25:19.580305 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpqm\" (UniqueName: \"kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm\") pod \"aca66151-f3fc-4be2-930e-946090928e89\" (UID: \"aca66151-f3fc-4be2-930e-946090928e89\") "
Sep 17 18:25:19.580552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aca66151-f3fc-4be2-930e-946090928e89" (UID: "aca66151-f3fc-4be2-930e-946090928e89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:19.580552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aca66151-f3fc-4be2-930e-946090928e89" (UID: "aca66151-f3fc-4be2-930e-946090928e89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:19.580683 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.580566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aca66151-f3fc-4be2-930e-946090928e89" (UID: "aca66151-f3fc-4be2-930e-946090928e89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:19.582165 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.582141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aca66151-f3fc-4be2-930e-946090928e89" (UID: "aca66151-f3fc-4be2-930e-946090928e89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:19.582247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.582201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm" (OuterVolumeSpecName: "kube-api-access-rqpqm") pod "aca66151-f3fc-4be2-930e-946090928e89" (UID: "aca66151-f3fc-4be2-930e-946090928e89"). InnerVolumeSpecName "kube-api-access-rqpqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:19.681318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.681246    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aca66151-f3fc-4be2-930e-946090928e89-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:19.681318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.681271    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:19.681318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.681280    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aca66151-f3fc-4be2-930e-946090928e89-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:19.681318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.681288    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpqm\" (UniqueName: \"kubernetes.io/projected/aca66151-f3fc-4be2-930e-946090928e89-kube-api-access-rqpqm\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:19.681318 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.681297    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aca66151-f3fc-4be2-930e-946090928e89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:19.973890 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.973790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerStarted","Data":"65882d55adf2050ff86e9a7f507c0223d01ba7daa055601b112f09fd7833d24b"}
Sep 17 18:25:19.975823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.975796    2577 generic.go:356] "Generic (PLEG): container finished" podID="aca66151-f3fc-4be2-930e-946090928e89" containerID="e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26" exitCode=0
Sep 17 18:25:19.975942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.975862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"
Sep 17 18:25:19.975942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.975880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerDied","Data":"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"}
Sep 17 18:25:19.975942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.975930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q" event={"ID":"aca66151-f3fc-4be2-930e-946090928e89","Type":"ContainerDied","Data":"7e76ef1b1ed959a4375ccc53ab7c50f1b9b796b2edba5f3d5ad3c507cfe49cab"}
Sep 17 18:25:19.976045 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.975959    2577 scope.go:117] "RemoveContainer" containerID="390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"
Sep 17 18:25:19.984553 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.984539    2577 scope.go:117] "RemoveContainer" containerID="e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"
Sep 17 18:25:19.992677 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:19.992657    2577 scope.go:117] "RemoveContainer" containerID="aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"
Sep 17 18:25:20.000080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000066    2577 scope.go:117] "RemoveContainer" containerID="390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"
Sep 17 18:25:20.000327 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:20.000307    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413\": container with ID starting with 390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413 not found: ID does not exist" containerID="390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"
Sep 17 18:25:20.000380 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413"} err="failed to get container status \"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413\": rpc error: code = NotFound desc = could not find container \"390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413\": container with ID starting with 390f780f9803c0d56e346cd079a88ce4431f1d1351b5508d6906bb7975fd0413 not found: ID does not exist"
Sep 17 18:25:20.000380 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000348    2577 scope.go:117] "RemoveContainer" containerID="e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"
Sep 17 18:25:20.000582 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:20.000566    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26\": container with ID starting with e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26 not found: ID does not exist" containerID="e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"
Sep 17 18:25:20.000621 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000584    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26"} err="failed to get container status \"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26\": rpc error: code = NotFound desc = could not find container \"e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26\": container with ID starting with e1fa58dbb5ff199218c944df130109d688e230bae7e9a484efe3b70226668d26 not found: ID does not exist"
Sep 17 18:25:20.000621 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000598    2577 scope.go:117] "RemoveContainer" containerID="aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"
Sep 17 18:25:20.000838 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:20.000823    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e\": container with ID starting with aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e not found: ID does not exist" containerID="aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"
Sep 17 18:25:20.000886 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.000840    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e"} err="failed to get container status \"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e\": rpc error: code = NotFound desc = could not find container \"aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e\": container with ID starting with aa2bef8eda5a869e85d0c842b56ec55fc36b0fe777877fd8b4469e62442e9f6e not found: ID does not exist"
Sep 17 18:25:20.005015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.004993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:25:20.009160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.009141    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qlc9q"]
Sep 17 18:25:20.941401 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:20.941370    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca66151-f3fc-4be2-930e-946090928e89" path="/var/lib/kubelet/pods/aca66151-f3fc-4be2-930e-946090928e89/volumes"
Sep 17 18:25:26.006727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:26.006642    2577 generic.go:356] "Generic (PLEG): container finished" podID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerID="65882d55adf2050ff86e9a7f507c0223d01ba7daa055601b112f09fd7833d24b" exitCode=0
Sep 17 18:25:26.006727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:26.006716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerDied","Data":"65882d55adf2050ff86e9a7f507c0223d01ba7daa055601b112f09fd7833d24b"}
Sep 17 18:25:27.014365 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:27.014327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerStarted","Data":"e5d63f309175cbfe71656c812955e5c957ecaf1568662287979ffe09a827194b"}
Sep 17 18:25:27.014789 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:27.014413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerStarted","Data":"359a5f0f8761a7b9af74a06405adf73219d2206a8da183095f4955a85b25c2a9"}
Sep 17 18:25:27.014789 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:27.014641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:27.038287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:27.038240    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podStartSLOduration=3.300521022 podStartE2EDuration="10.038224761s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.737703739s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:27.036376435 +0000 UTC m=+1738.360697862" watchObservedRunningTime="2026-09-17 18:25:27.038224761 +0000 UTC m=+1738.362546252"
Sep 17 18:25:28.020146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:28.020112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:28.021391 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:28.021363    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:25:29.024634 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:29.024595    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:25:34.029980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:34.029954    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:34.030619 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:34.030592    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:25:39.069951 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:39.069915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:44.031345 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:44.031315    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:44.204536 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:44.204508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:44.445807 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:44.445778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:50.198769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:50.198722    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:50.199159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:50.199026    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container" containerID="cri-o://359a5f0f8761a7b9af74a06405adf73219d2206a8da183095f4955a85b25c2a9" gracePeriod=30
Sep 17 18:25:50.199159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:50.199081    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kube-rbac-proxy" containerID="cri-o://e5d63f309175cbfe71656c812955e5c957ecaf1568662287979ffe09a827194b" gracePeriod=30
Sep 17 18:25:50.216338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:50.216312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:51.139009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.138977    2577 generic.go:356] "Generic (PLEG): container finished" podID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerID="e5d63f309175cbfe71656c812955e5c957ecaf1568662287979ffe09a827194b" exitCode=2
Sep 17 18:25:51.139009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.139003    2577 generic.go:356] "Generic (PLEG): container finished" podID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerID="359a5f0f8761a7b9af74a06405adf73219d2206a8da183095f4955a85b25c2a9" exitCode=0
Sep 17 18:25:51.139188 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.139034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerDied","Data":"e5d63f309175cbfe71656c812955e5c957ecaf1568662287979ffe09a827194b"}
Sep 17 18:25:51.139188 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.139091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerDied","Data":"359a5f0f8761a7b9af74a06405adf73219d2206a8da183095f4955a85b25c2a9"}
Sep 17 18:25:51.234127 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.234105    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:51.350432 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location\") pod \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") "
Sep 17 18:25:51.350593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350448    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") pod \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") "
Sep 17 18:25:51.350593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config\") pod \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") "
Sep 17 18:25:51.350593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-667df\" (UniqueName: \"kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df\") pod \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") "
Sep 17 18:25:51.350593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert\") pod \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\" (UID: \"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116\") "
Sep 17 18:25:51.350901 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:51.350968 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.350947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:51.352638 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.352608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df" (OuterVolumeSpecName: "kube-api-access-667df") pod "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116"). InnerVolumeSpecName "kube-api-access-667df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:51.352638 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.352624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:51.375578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.375553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" (UID: "6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:51.451591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.451566    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:51.451591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.451586    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:51.451591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.451595    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:51.451789 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.451604    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667df\" (UniqueName: \"kubernetes.io/projected/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-kube-api-access-667df\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:51.451789 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:51.451612    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:25:52.145587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.145551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw" event={"ID":"6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116","Type":"ContainerDied","Data":"2850aeaf2a0bb802270682dec1bd2ea18bbca605cffb56f94877873135bbc62a"}
Sep 17 18:25:52.145791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.145617    2577 scope.go:117] "RemoveContainer" containerID="e5d63f309175cbfe71656c812955e5c957ecaf1568662287979ffe09a827194b"
Sep 17 18:25:52.145791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.145565    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"
Sep 17 18:25:52.154510 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.154492    2577 scope.go:117] "RemoveContainer" containerID="359a5f0f8761a7b9af74a06405adf73219d2206a8da183095f4955a85b25c2a9"
Sep 17 18:25:52.161697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.161681    2577 scope.go:117] "RemoveContainer" containerID="65882d55adf2050ff86e9a7f507c0223d01ba7daa055601b112f09fd7833d24b"
Sep 17 18:25:52.166643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.166622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:52.170996 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.170974    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-577d78f5bd-77cjw"]
Sep 17 18:25:52.942371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:52.942328    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" path="/var/lib/kubelet/pods/6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116/volumes"
Sep 17 18:25:55.366189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.366157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:55.391580 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.391542    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:25:55.391964 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.391947    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kube-rbac-proxy"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.391967    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kube-rbac-proxy"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.391991    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kserve-container"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.391999    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kserve-container"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392021    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392033    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="storage-initializer"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392041    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="storage-initializer"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392058    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="storage-initializer"
Sep 17 18:25:55.392075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392066    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="storage-initializer"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kube-rbac-proxy"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392095    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kube-rbac-proxy"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392177    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kube-rbac-proxy"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392191    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kserve-container"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392200    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de1b1b0-cce2-49c7-b2e6-aedcdcbfc116" containerName="kserve-container"
Sep 17 18:25:55.392531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.392210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca66151-f3fc-4be2-930e-946090928e89" containerName="kube-rbac-proxy"
Sep 17 18:25:55.395589 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.395560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.397842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.397820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 18:25:55.397973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.397953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:25:55.397973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.397970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:25:55.398080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.398061    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:25:55.398159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.398146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:25:55.398290 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.398277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:25:55.398820 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.398806    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:25:55.406131 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.406112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:25:55.483866 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.483833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvhh\" (UniqueName: \"kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.484035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.483873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.484035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.483910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.484035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.483930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.484035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.483951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584592 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.584554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584592 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.584598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584883 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.584627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584883 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.584694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvhh\" (UniqueName: \"kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584883 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.584732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.584883 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:55.584820    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 18:25:55.585065 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:25:55.584901    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls podName:ec3e28e7-e416-40d2-8e9f-189965bc710e nodeName:}" failed. No retries permitted until 2026-09-17 18:25:56.084882601 +0000 UTC m=+1767.409204003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 18:25:55.585117 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.585090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.585298 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.585282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.585371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.585352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.595152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.595131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvhh\" (UniqueName: \"kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:55.708278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:55.708205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:56.089017 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:56.088932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:56.091359 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:56.091342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:56.307020 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:56.306988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:25:56.580463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:56.580440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:25:56.582627 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:25:56.582598    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3e28e7_e416_40d2_8e9f_189965bc710e.slice/crio-02d11f8715bbea249debad12f2a50de91954a8ee5be7ae16f61c7b6398495027 WatchSource:0}: Error finding container 02d11f8715bbea249debad12f2a50de91954a8ee5be7ae16f61c7b6398495027: Status 404 returned error can't find the container with id 02d11f8715bbea249debad12f2a50de91954a8ee5be7ae16f61c7b6398495027
Sep 17 18:25:57.171060 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:57.171024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerStarted","Data":"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"}
Sep 17 18:25:57.171247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:25:57.171088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerStarted","Data":"02d11f8715bbea249debad12f2a50de91954a8ee5be7ae16f61c7b6398495027"}
Sep 17 18:26:01.193447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:01.193405    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerID="ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073" exitCode=0
Sep 17 18:26:01.193912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:01.193481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerDied","Data":"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"}
Sep 17 18:26:02.200711 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:02.200678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerStarted","Data":"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"}
Sep 17 18:26:02.201159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:02.200736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerStarted","Data":"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"}
Sep 17 18:26:02.201159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:02.200967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:26:02.201159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:02.201014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:26:02.222767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:02.222705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" podStartSLOduration=3.342583735 podStartE2EDuration="7.222689587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880105852s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:02.221204358 +0000 UTC m=+1773.545525789" watchObservedRunningTime="2026-09-17 18:26:02.222689587 +0000 UTC m=+1773.547011011"
Sep 17 18:26:08.211596 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:08.211569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:26:19.662173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:19.662134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:24.793505 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.793469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:24.840238 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.840209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:26:24.843817 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.843799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.846134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.846114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:26:24.846251 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.846129    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:26:24.855423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.855402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:26:24.873448 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.873421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:24.878231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.878212    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:26:24.881626 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.881597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.884825 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.884287    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:26:24.893527 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.893503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:26:24.947331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.947466 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.947466 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.947466 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fcs\" (UniqueName: \"kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.947591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.947591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.947591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.947591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.947591 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:24.947795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.947604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrfr\" (UniqueName: \"kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:24.953975 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:24.953952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:25.048157 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fcs\" (UniqueName: \"kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.048157 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.048157 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrfr\" (UniqueName: \"kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:26:25.048311    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:26:25.048390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls podName:ab22b72b-3c59-40f8-90cc-9660eb06e318 nodeName:}" failed. No retries permitted until 2026-09-17 18:26:25.548369089 +0000 UTC m=+1796.872690504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls") pod "isvc-canary-create-v2-predictor-5bb585469b-z7m5g" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 18:26:25.048440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048950 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.048950 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.048860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.049114 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.049089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.049224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.049095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.049224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.049134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.049443 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.049425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.050728 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.050707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.057973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.057952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fcs\" (UniqueName: \"kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs\") pod \"isvc-canary-create-predictor-5956b77f85-7gqlg\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.059436 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.059414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrfr\" (UniqueName: \"kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.198872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.198837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:25.312203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.312129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:25.472719 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.472692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:26:25.474895 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:26:25.474873    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027c91b8_156b_425f_855d_06745357ecb1.slice/crio-eebd8fa21c87c402e6e0a949b06f5f72a0711039c4b1d3895a61521131c63d42 WatchSource:0}: Error finding container eebd8fa21c87c402e6e0a949b06f5f72a0711039c4b1d3895a61521131c63d42: Status 404 returned error can't find the container with id eebd8fa21c87c402e6e0a949b06f5f72a0711039c4b1d3895a61521131c63d42
Sep 17 18:26:25.553334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.553309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.555579 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.555541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-z7m5g\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:25.755031 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:25.754998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:26.032649 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:26.032621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:26:26.033813 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:26:26.033786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab22b72b_3c59_40f8_90cc_9660eb06e318.slice/crio-25234c0ea8d7e8b49705d15336d4c3266519a089771fdec5d7f94e2a18768695 WatchSource:0}: Error finding container 25234c0ea8d7e8b49705d15336d4c3266519a089771fdec5d7f94e2a18768695: Status 404 returned error can't find the container with id 25234c0ea8d7e8b49705d15336d4c3266519a089771fdec5d7f94e2a18768695
Sep 17 18:26:26.315209 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:26.315115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerStarted","Data":"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"}
Sep 17 18:26:26.315209 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:26.315175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerStarted","Data":"eebd8fa21c87c402e6e0a949b06f5f72a0711039c4b1d3895a61521131c63d42"}
Sep 17 18:26:26.316806 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:26.316781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerStarted","Data":"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"}
Sep 17 18:26:26.316918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:26.316833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerStarted","Data":"25234c0ea8d7e8b49705d15336d4c3266519a089771fdec5d7f94e2a18768695"}
Sep 17 18:26:29.332701 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:29.332668    2577 generic.go:356] "Generic (PLEG): container finished" podID="027c91b8-156b-425f-855d-06745357ecb1" containerID="e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7" exitCode=0
Sep 17 18:26:29.333089 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:29.332767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerDied","Data":"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"}
Sep 17 18:26:30.345484 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.345399    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerID="69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1" exitCode=0
Sep 17 18:26:30.345484 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.345472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerDied","Data":"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"}
Sep 17 18:26:30.347776 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.347738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerStarted","Data":"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"}
Sep 17 18:26:30.347879 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.347808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerStarted","Data":"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"}
Sep 17 18:26:30.348028 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.348009    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:30.384178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:30.384122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podStartSLOduration=2.622189902 podStartE2EDuration="6.384108107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761918205s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:30.383181943 +0000 UTC m=+1801.707503380" watchObservedRunningTime="2026-09-17 18:26:30.384108107 +0000 UTC m=+1801.708429530"
Sep 17 18:26:31.356036 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.356000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerStarted","Data":"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"}
Sep 17 18:26:31.356455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.356071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:31.356455 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.356101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerStarted","Data":"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"}
Sep 17 18:26:31.356588 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.356568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:31.356649 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.356618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:31.357297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.357269    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:26:31.357397 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.357310    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:26:31.375716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:31.375661    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podStartSLOduration=3.6186469 podStartE2EDuration="7.375642828s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756995928s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:31.373164042 +0000 UTC m=+1802.697485466" watchObservedRunningTime="2026-09-17 18:26:31.375642828 +0000 UTC m=+1802.699964253"
Sep 17 18:26:32.360559 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:32.360517    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:26:32.360982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:32.360619    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:26:37.366255 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:37.366223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:26:37.366784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:37.366316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:26:37.366784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:37.366720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:26:37.366784 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:37.366732    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:26:38.272480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:38.272428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:26:47.366854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:47.366813    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:26:47.367296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:47.366813    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:26:48.214160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:48.214134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:26:55.450602 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:55.450568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:26:55.451057 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:55.450822    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kserve-container" containerID="cri-o://b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a" gracePeriod=30
Sep 17 18:26:55.451057 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:55.450857    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kube-rbac-proxy" containerID="cri-o://380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6" gracePeriod=30
Sep 17 18:26:55.466024 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:55.465994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:56.484359 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:56.484326    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerID="380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6" exitCode=2
Sep 17 18:26:56.484793 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:56.484400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerDied","Data":"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"}
Sep 17 18:26:57.367264 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:57.367227    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:26:57.367439 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:57.367232    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:26:58.206275 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:26:58.206238    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 18:27:00.612420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:00.612385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:00.754767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:00.754733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:02.905487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.905461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:27:02.993719 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.993618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location\") pod \"ec3e28e7-e416-40d2-8e9f-189965bc710e\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") "
Sep 17 18:27:02.993719 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.993669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") pod \"ec3e28e7-e416-40d2-8e9f-189965bc710e\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") "
Sep 17 18:27:02.993988 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.993737    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert\") pod \"ec3e28e7-e416-40d2-8e9f-189965bc710e\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") "
Sep 17 18:27:02.993988 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.993788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config\") pod \"ec3e28e7-e416-40d2-8e9f-189965bc710e\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") "
Sep 17 18:27:02.993988 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.993833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzvhh\" (UniqueName: \"kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh\") pod \"ec3e28e7-e416-40d2-8e9f-189965bc710e\" (UID: \"ec3e28e7-e416-40d2-8e9f-189965bc710e\") "
Sep 17 18:27:02.994138 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.994084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec3e28e7-e416-40d2-8e9f-189965bc710e" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:02.994138 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.994119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec3e28e7-e416-40d2-8e9f-189965bc710e" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:02.994217 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.994180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec3e28e7-e416-40d2-8e9f-189965bc710e" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:02.995948 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.995924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh" (OuterVolumeSpecName: "kube-api-access-bzvhh") pod "ec3e28e7-e416-40d2-8e9f-189965bc710e" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e"). InnerVolumeSpecName "kube-api-access-bzvhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:02.996067 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:02.995980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec3e28e7-e416-40d2-8e9f-189965bc710e" (UID: "ec3e28e7-e416-40d2-8e9f-189965bc710e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:03.094809 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.094744    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec3e28e7-e416-40d2-8e9f-189965bc710e-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:03.094809 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.094804    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec3e28e7-e416-40d2-8e9f-189965bc710e-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:03.094809 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.094813    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:03.095004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.094824    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:03.095004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.094836    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvhh\" (UniqueName: \"kubernetes.io/projected/ec3e28e7-e416-40d2-8e9f-189965bc710e-kube-api-access-bzvhh\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:03.519809 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.519772    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerID="b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a" exitCode=0
Sep 17 18:27:03.519984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.519822    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"
Sep 17 18:27:03.519984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.519825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerDied","Data":"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"}
Sep 17 18:27:03.519984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.519883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2" event={"ID":"ec3e28e7-e416-40d2-8e9f-189965bc710e","Type":"ContainerDied","Data":"02d11f8715bbea249debad12f2a50de91954a8ee5be7ae16f61c7b6398495027"}
Sep 17 18:27:03.519984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.519912    2577 scope.go:117] "RemoveContainer" containerID="380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"
Sep 17 18:27:03.528360 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.528338    2577 scope.go:117] "RemoveContainer" containerID="b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"
Sep 17 18:27:03.536001 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.535982    2577 scope.go:117] "RemoveContainer" containerID="ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"
Sep 17 18:27:03.541727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.541701    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:27:03.544467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.544450    2577 scope.go:117] "RemoveContainer" containerID="380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"
Sep 17 18:27:03.544776 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:03.544745    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6\": container with ID starting with 380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6 not found: ID does not exist" containerID="380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"
Sep 17 18:27:03.544860 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.544788    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6"} err="failed to get container status \"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6\": rpc error: code = NotFound desc = could not find container \"380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6\": container with ID starting with 380e50a597f8b9481c2a84d723307de7dee809b91a5878901806caba00d489a6 not found: ID does not exist"
Sep 17 18:27:03.544860 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.544814    2577 scope.go:117] "RemoveContainer" containerID="b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"
Sep 17 18:27:03.545081 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:03.545064    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a\": container with ID starting with b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a not found: ID does not exist" containerID="b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"
Sep 17 18:27:03.545149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.545089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a"} err="failed to get container status \"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a\": rpc error: code = NotFound desc = could not find container \"b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a\": container with ID starting with b2dd9ca87f5bbe8aa9a95fb1cbbcc26a66cbcb849615e160b90fb1c5b16abc2a not found: ID does not exist"
Sep 17 18:27:03.545149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.545110    2577 scope.go:117] "RemoveContainer" containerID="ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"
Sep 17 18:27:03.545307 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.545288    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lh9g2"]
Sep 17 18:27:03.545388 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:03.545372    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073\": container with ID starting with ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073 not found: ID does not exist" containerID="ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"
Sep 17 18:27:03.545432 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:03.545394    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073"} err="failed to get container status \"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073\": rpc error: code = NotFound desc = could not find container \"ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073\": container with ID starting with ad3f5d7f9c447acaeffa26356d23a1a893c3cc97f22aa1858e512de61bd8e073 not found: ID does not exist"
Sep 17 18:27:04.941641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:04.941599    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" path="/var/lib/kubelet/pods/ec3e28e7-e416-40d2-8e9f-189965bc710e/volumes"
Sep 17 18:27:07.366959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:07.366925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:27:07.367386 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:07.366929    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:27:16.509739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:16.509636    2577 scope.go:117] "RemoveContainer" containerID="4c2c706673d16b88b6e71a0bfebd70b67bf733df92425c262dded712e542faa0"
Sep 17 18:27:16.529255 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:16.519424    2577 scope.go:117] "RemoveContainer" containerID="5ba0a1ed780b31a58336cb19beda3fc001f06e0c3105fcc651f515ed93f30665"
Sep 17 18:27:17.366987 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:17.366943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:27:17.367288 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:17.366943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:27:27.367478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:27.367432    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:27:27.367954 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:27.367442    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:27:37.367265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:37.367227    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:27:37.367656 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:37.367228    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:27:47.368059 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:47.368025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:27:47.368505 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:47.368133    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:27:54.880564 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.880534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:27:54.881161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.880870    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" containerID="cri-o://3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db" gracePeriod=30
Sep 17 18:27:54.881161 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.880907    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kube-rbac-proxy" containerID="cri-o://bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c" gracePeriod=30
Sep 17 18:27:54.885028 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.885003    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:27:54.885248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.885216    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" containerID="cri-o://2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6" gracePeriod=30
Sep 17 18:27:54.885360 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.885313    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kube-rbac-proxy" containerID="cri-o://4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b" gracePeriod=30
Sep 17 18:27:54.920369 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:54.920348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:55.001353 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:55.001328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:55.753151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:55.753122    2577 generic.go:356] "Generic (PLEG): container finished" podID="027c91b8-156b-425f-855d-06745357ecb1" containerID="4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b" exitCode=2
Sep 17 18:27:55.753333 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:55.753195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerDied","Data":"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"}
Sep 17 18:27:55.755164 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:55.755144    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerID="bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c" exitCode=2
Sep 17 18:27:55.755268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:55.755199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerDied","Data":"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"}
Sep 17 18:27:57.360720 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:57.360676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 18:27:57.361171 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:57.360686    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 18:27:57.367117 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:57.367092    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:27:57.367117 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:57.367104    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:27:59.548160 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.548139    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:27:59.550868 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.550851    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:27:59.672398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") pod \"ab22b72b-3c59-40f8-90cc-9660eb06e318\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") "
Sep 17 18:27:59.672398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls\") pod \"027c91b8-156b-425f-855d-06745357ecb1\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") "
Sep 17 18:27:59.672398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srrfr\" (UniqueName: \"kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr\") pod \"ab22b72b-3c59-40f8-90cc-9660eb06e318\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") "
Sep 17 18:27:59.672398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8fcs\" (UniqueName: \"kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs\") pod \"027c91b8-156b-425f-855d-06745357ecb1\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert\") pod \"ab22b72b-3c59-40f8-90cc-9660eb06e318\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location\") pod \"ab22b72b-3c59-40f8-90cc-9660eb06e318\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config\") pod \"027c91b8-156b-425f-855d-06745357ecb1\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672557    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location\") pod \"027c91b8-156b-425f-855d-06745357ecb1\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config\") pod \"ab22b72b-3c59-40f8-90cc-9660eb06e318\" (UID: \"ab22b72b-3c59-40f8-90cc-9660eb06e318\") "
Sep 17 18:27:59.672732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert\") pod \"027c91b8-156b-425f-855d-06745357ecb1\" (UID: \"027c91b8-156b-425f-855d-06745357ecb1\") "
Sep 17 18:27:59.673079 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab22b72b-3c59-40f8-90cc-9660eb06e318" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:59.673079 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.672992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "027c91b8-156b-425f-855d-06745357ecb1" (UID: "027c91b8-156b-425f-855d-06745357ecb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:59.673189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.673112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab22b72b-3c59-40f8-90cc-9660eb06e318" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:59.673329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.673303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "027c91b8-156b-425f-855d-06745357ecb1" (UID: "027c91b8-156b-425f-855d-06745357ecb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:59.673498 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.673305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "027c91b8-156b-425f-855d-06745357ecb1" (UID: "027c91b8-156b-425f-855d-06745357ecb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:59.673498 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.673364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab22b72b-3c59-40f8-90cc-9660eb06e318" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:59.674746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.674717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs" (OuterVolumeSpecName: "kube-api-access-k8fcs") pod "027c91b8-156b-425f-855d-06745357ecb1" (UID: "027c91b8-156b-425f-855d-06745357ecb1"). InnerVolumeSpecName "kube-api-access-k8fcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:59.674846 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.674800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "027c91b8-156b-425f-855d-06745357ecb1" (UID: "027c91b8-156b-425f-855d-06745357ecb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:59.675048 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.675028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab22b72b-3c59-40f8-90cc-9660eb06e318" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:59.675048 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.675036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr" (OuterVolumeSpecName: "kube-api-access-srrfr") pod "ab22b72b-3c59-40f8-90cc-9660eb06e318" (UID: "ab22b72b-3c59-40f8-90cc-9660eb06e318"). InnerVolumeSpecName "kube-api-access-srrfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:59.773659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773628    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab22b72b-3c59-40f8-90cc-9660eb06e318-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773659    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/027c91b8-156b-425f-855d-06745357ecb1-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773672    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srrfr\" (UniqueName: \"kubernetes.io/projected/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-api-access-srrfr\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fcs\" (UniqueName: \"kubernetes.io/projected/027c91b8-156b-425f-855d-06745357ecb1-kube-api-access-k8fcs\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773697    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773708    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab22b72b-3c59-40f8-90cc-9660eb06e318-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773720    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773732    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/027c91b8-156b-425f-855d-06745357ecb1-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773744    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab22b72b-3c59-40f8-90cc-9660eb06e318-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.773922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.773806    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/027c91b8-156b-425f-855d-06745357ecb1-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:27:59.779005 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.778977    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerID="3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db" exitCode=0
Sep 17 18:27:59.779142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.779006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerDied","Data":"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"}
Sep 17 18:27:59.779142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.779031    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"
Sep 17 18:27:59.779142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.779071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g" event={"ID":"ab22b72b-3c59-40f8-90cc-9660eb06e318","Type":"ContainerDied","Data":"25234c0ea8d7e8b49705d15336d4c3266519a089771fdec5d7f94e2a18768695"}
Sep 17 18:27:59.779142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.779097    2577 scope.go:117] "RemoveContainer" containerID="bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"
Sep 17 18:27:59.781079 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.781051    2577 generic.go:356] "Generic (PLEG): container finished" podID="027c91b8-156b-425f-855d-06745357ecb1" containerID="2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6" exitCode=0
Sep 17 18:27:59.781163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.781112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerDied","Data":"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"}
Sep 17 18:27:59.781163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.781118    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"
Sep 17 18:27:59.781163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.781153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg" event={"ID":"027c91b8-156b-425f-855d-06745357ecb1","Type":"ContainerDied","Data":"eebd8fa21c87c402e6e0a949b06f5f72a0711039c4b1d3895a61521131c63d42"}
Sep 17 18:27:59.787995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.787975    2577 scope.go:117] "RemoveContainer" containerID="3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"
Sep 17 18:27:59.796240 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.796223    2577 scope.go:117] "RemoveContainer" containerID="69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"
Sep 17 18:27:59.806113 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.804697    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:27:59.806280 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.806180    2577 scope.go:117] "RemoveContainer" containerID="bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"
Sep 17 18:27:59.806680 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.806659    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c\": container with ID starting with bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c not found: ID does not exist" containerID="bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"
Sep 17 18:27:59.806787 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.806687    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c"} err="failed to get container status \"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c\": rpc error: code = NotFound desc = could not find container \"bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c\": container with ID starting with bd42b9e8430bc28fc62c9cf825c0a365380df3379c3d343117b91c5b1574a60c not found: ID does not exist"
Sep 17 18:27:59.806787 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.806715    2577 scope.go:117] "RemoveContainer" containerID="3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"
Sep 17 18:27:59.807042 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.807016    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db\": container with ID starting with 3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db not found: ID does not exist" containerID="3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"
Sep 17 18:27:59.807145 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.807048    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db"} err="failed to get container status \"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db\": rpc error: code = NotFound desc = could not find container \"3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db\": container with ID starting with 3b28263b3a34156b86dd98f926dabbcf0814d248d7b1a35b42e42e47533482db not found: ID does not exist"
Sep 17 18:27:59.807145 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.807066    2577 scope.go:117] "RemoveContainer" containerID="69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"
Sep 17 18:27:59.807665 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.807518    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1\": container with ID starting with 69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1 not found: ID does not exist" containerID="69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"
Sep 17 18:27:59.807665 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.807606    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1"} err="failed to get container status \"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1\": rpc error: code = NotFound desc = could not find container \"69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1\": container with ID starting with 69cd9e601c5d3b4b79dfb471876ab53fda75ec96d92e975d53f895054f522cb1 not found: ID does not exist"
Sep 17 18:27:59.807665 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.807632    2577 scope.go:117] "RemoveContainer" containerID="4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"
Sep 17 18:27:59.809350 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.809329    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-z7m5g"]
Sep 17 18:27:59.815215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.815198    2577 scope.go:117] "RemoveContainer" containerID="2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"
Sep 17 18:27:59.819649 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.819627    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:27:59.822793 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.822770    2577 scope.go:117] "RemoveContainer" containerID="e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"
Sep 17 18:27:59.823126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.823111    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-7gqlg"]
Sep 17 18:27:59.829965 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.829944    2577 scope.go:117] "RemoveContainer" containerID="4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"
Sep 17 18:27:59.830238 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.830213    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b\": container with ID starting with 4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b not found: ID does not exist" containerID="4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"
Sep 17 18:27:59.830281 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.830244    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b"} err="failed to get container status \"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b\": rpc error: code = NotFound desc = could not find container \"4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b\": container with ID starting with 4c131fe323b807c320461c0c4470f670943a78bce922a0795f86432df92b280b not found: ID does not exist"
Sep 17 18:27:59.830281 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.830262    2577 scope.go:117] "RemoveContainer" containerID="2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"
Sep 17 18:27:59.830505 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.830489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6\": container with ID starting with 2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6 not found: ID does not exist" containerID="2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"
Sep 17 18:27:59.830554 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.830510    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6"} err="failed to get container status \"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6\": rpc error: code = NotFound desc = could not find container \"2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6\": container with ID starting with 2c35f093256cdc79aa44c2a1679c68dbbd887fb0be8a122c0b4647e2858d55b6 not found: ID does not exist"
Sep 17 18:27:59.830554 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.830534    2577 scope.go:117] "RemoveContainer" containerID="e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"
Sep 17 18:27:59.830742 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:27:59.830726    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7\": container with ID starting with e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7 not found: ID does not exist" containerID="e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"
Sep 17 18:27:59.830801 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:27:59.830746    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7"} err="failed to get container status \"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7\": rpc error: code = NotFound desc = could not find container \"e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7\": container with ID starting with e2745947872cfcff457cf688c04beb03acef2bf44e7760f1ea80f39a47f68dd7 not found: ID does not exist"
Sep 17 18:28:00.065555 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065480    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:28:00.065824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kserve-container"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065826    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kserve-container"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065836    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="storage-initializer"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="storage-initializer"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065849    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kube-rbac-proxy"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065856    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kube-rbac-proxy"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065864    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="storage-initializer"
Sep 17 18:28:00.065875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065869    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="storage-initializer"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065877    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="storage-initializer"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065883    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="storage-initializer"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065891    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065896    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065912    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065938    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065943    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.065994    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.066001    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.066009    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kserve-container"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.066015    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="027c91b8-156b-425f-855d-06745357ecb1" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.066021    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3e28e7-e416-40d2-8e9f-189965bc710e" containerName="kube-rbac-proxy"
Sep 17 18:28:00.066125 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.066027    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" containerName="kube-rbac-proxy"
Sep 17 18:28:00.071956 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.071939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.074423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:28:00.074423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:28:00.074747 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:28:00.074855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:28:00.074855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:28:00.074855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:28:00.074855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.074843    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:28:00.079938 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.079915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:28:00.084918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.084890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.108167 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.108140    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:28:00.111304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.111282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.113536 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.113516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:28:00.121541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.121520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:28:00.178507 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqztt\" (UniqueName: \"kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.178682 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.178682 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.178819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.178819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.178819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.178819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.179002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.179002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.179002 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.178931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2rc\" (UniqueName: \"kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.180004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.179962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.279930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.279894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.279930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.279931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.279954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2rc\" (UniqueName: \"kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqztt\" (UniqueName: \"kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280518 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:28:00.280344    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:28:00.280518 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:28:00.280410    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls podName:b8949bcd-771a-4acf-914e-b827ba13728c nodeName:}" failed. No retries permitted until 2026-09-17 18:28:00.780387714 +0000 UTC m=+1892.104709134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls") pod "isvc-canary-promote-predictor-7877c757df-5bqqs" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:28:00.280518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.280795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.280971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.281010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.280994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.282627 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.282611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.289039 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.289020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqztt\" (UniqueName: \"kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.289216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.289200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2rc\" (UniqueName: \"kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.383715 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.383685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:00.404332 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.404303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.656743 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.656717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:28:00.659017 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:28:00.658993    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cc4bf1_46cd_4152_82be_b9a5d4ed5f2b.slice/crio-f7ede3f4d80dd6093ae2678c0791f08d15d68f6732e70ad42efb6a0d91e1e42f WatchSource:0}: Error finding container f7ede3f4d80dd6093ae2678c0791f08d15d68f6732e70ad42efb6a0d91e1e42f: Status 404 returned error can't find the container with id f7ede3f4d80dd6093ae2678c0791f08d15d68f6732e70ad42efb6a0d91e1e42f
Sep 17 18:28:00.784223 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.784194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.786920 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.786886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-5bqqs\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:00.789238 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.789205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerStarted","Data":"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"}
Sep 17 18:28:00.789372 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.789277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerStarted","Data":"f7ede3f4d80dd6093ae2678c0791f08d15d68f6732e70ad42efb6a0d91e1e42f"}
Sep 17 18:28:00.947742 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.947664    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027c91b8-156b-425f-855d-06745357ecb1" path="/var/lib/kubelet/pods/027c91b8-156b-425f-855d-06745357ecb1/volumes"
Sep 17 18:28:00.948194 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:00.948178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab22b72b-3c59-40f8-90cc-9660eb06e318" path="/var/lib/kubelet/pods/ab22b72b-3c59-40f8-90cc-9660eb06e318/volumes"
Sep 17 18:28:01.021884 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:01.021845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:01.293878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:01.293805    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:28:01.296659 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:28:01.296635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8949bcd_771a_4acf_914e_b827ba13728c.slice/crio-06dfee3c84476ab84e0b5ce0fd83a866472d044b6d31405294a1191ca724f713 WatchSource:0}: Error finding container 06dfee3c84476ab84e0b5ce0fd83a866472d044b6d31405294a1191ca724f713: Status 404 returned error can't find the container with id 06dfee3c84476ab84e0b5ce0fd83a866472d044b6d31405294a1191ca724f713
Sep 17 18:28:01.796937 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:01.796893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerStarted","Data":"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"}
Sep 17 18:28:01.797428 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:01.796968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerStarted","Data":"06dfee3c84476ab84e0b5ce0fd83a866472d044b6d31405294a1191ca724f713"}
Sep 17 18:28:04.813061 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:04.812987    2577 generic.go:356] "Generic (PLEG): container finished" podID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerID="663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508" exitCode=0
Sep 17 18:28:04.813061 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:04.813053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerDied","Data":"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"}
Sep 17 18:28:05.820522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.820486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerStarted","Data":"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"}
Sep 17 18:28:05.820973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.820571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerStarted","Data":"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"}
Sep 17 18:28:05.820973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.820866    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:05.821080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.821007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:05.822434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.822409    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8949bcd-771a-4acf-914e-b827ba13728c" containerID="0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f" exitCode=0
Sep 17 18:28:05.822540 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.822457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerDied","Data":"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"}
Sep 17 18:28:05.822540 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.822456    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:05.840080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:05.840032    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podStartSLOduration=2.153528567 podStartE2EDuration="5.840013265s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.686484698s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:05.838683069 +0000 UTC m=+1897.163004492" watchObservedRunningTime="2026-09-17 18:28:05.840013265 +0000 UTC m=+1897.164334691"
Sep 17 18:28:06.830278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.830234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerStarted","Data":"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"}
Sep 17 18:28:06.830790 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.830311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerStarted","Data":"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"}
Sep 17 18:28:06.830790 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.830473    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:06.830790 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.830618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:06.830790 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.830648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:06.832096 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.832072    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:06.851010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:06.850965    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podStartSLOduration=3.162807559 podStartE2EDuration="6.850952485s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688144926s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:06.850216723 +0000 UTC m=+1898.174538146" watchObservedRunningTime="2026-09-17 18:28:06.850952485 +0000 UTC m=+1898.175273908"
Sep 17 18:28:07.835662 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:07.835622    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:11.834976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:11.834943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:28:11.835605 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:11.835575    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:12.840466 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:12.840440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:28:12.840973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:12.840947    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:21.836367 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:21.836330    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:22.841907 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:22.841870    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:30.813059 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:30.813005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:31.836425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:31.836382    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:32.840902 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:32.840865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:36.014290 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.014248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:36.075792 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.075742    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:28:36.080445 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.080428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.083587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.083569    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 18:28:36.083587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.083576    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 18:28:36.087538 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.087517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.087631 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.087552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp27p\" (UniqueName: \"kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.087696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.087635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.087696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.087667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.087834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.087696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.092270 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.092247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:28:36.188478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.188478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp27p\" (UniqueName: \"kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.188736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.188736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.188736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.188736 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:28:36.188719    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:28:36.188985 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:28:36.188819    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls podName:994d8bc3-4f09-4d4e-b765-49299ab7f05a nodeName:}" failed. No retries permitted until 2026-09-17 18:28:36.688802594 +0000 UTC m=+1928.013123996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:28:36.188985 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.188948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.189338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.189317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.189378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.189317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.199697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.199670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp27p\" (UniqueName: \"kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.256167 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.256139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:36.692629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.692588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.695028 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.695002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:36.991025 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:36.990941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:37.263391 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:37.263363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:28:37.264450 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:28:37.264417    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994d8bc3_4f09_4d4e_b765_49299ab7f05a.slice/crio-ffe1251ec0a63f024aa19874a7e26fede082a6a06cee270a20bcf8468cf07286 WatchSource:0}: Error finding container ffe1251ec0a63f024aa19874a7e26fede082a6a06cee270a20bcf8468cf07286: Status 404 returned error can't find the container with id ffe1251ec0a63f024aa19874a7e26fede082a6a06cee270a20bcf8468cf07286
Sep 17 18:28:37.974113 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:37.974076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerStarted","Data":"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"}
Sep 17 18:28:37.974360 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:37.974136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerStarted","Data":"ffe1251ec0a63f024aa19874a7e26fede082a6a06cee270a20bcf8468cf07286"}
Sep 17 18:28:41.836367 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:41.836331    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:41.995384 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:41.995351    2577 generic.go:356] "Generic (PLEG): container finished" podID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerID="31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff" exitCode=0
Sep 17 18:28:41.995552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:41.995417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerDied","Data":"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"}
Sep 17 18:28:42.841739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:42.841701    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:28:43.002769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.002725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerStarted","Data":"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"}
Sep 17 18:28:43.002935 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.002799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerStarted","Data":"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"}
Sep 17 18:28:43.003084 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.003068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:43.003264 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.003241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:43.004369 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.004338    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:28:43.022468 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:43.022430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podStartSLOduration=3.280471366 podStartE2EDuration="7.022412587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741941221s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:43.021420112 +0000 UTC m=+1934.345741537" watchObservedRunningTime="2026-09-17 18:28:43.022412587 +0000 UTC m=+1934.346734014"
Sep 17 18:28:44.008616 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:44.008577    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:28:45.013065 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:45.013032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:28:50.017271 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:50.017241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:28:50.017730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:50.017704    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:28:51.836304 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:51.836266    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:28:52.841885 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:28:52.841848    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:29:00.017794 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:00.017731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:01.835869 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:01.835834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:29:02.841371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:02.841335    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:29:10.018391 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:10.018353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:11.836007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:11.835967    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:29:12.840911 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:12.840833    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:29:20.018250 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:20.018205    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:21.836490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:21.836461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:29:21.958985 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:21.958951    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:29:21.959244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:21.959219    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container" containerID="cri-o://868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593" gracePeriod=30
Sep 17 18:29:21.959336 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:21.959272    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kube-rbac-proxy" containerID="cri-o://a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a" gracePeriod=30
Sep 17 18:29:21.960088 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:21.959929    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:29:22.197600 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:22.197569    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8949bcd-771a-4acf-914e-b827ba13728c" containerID="a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a" exitCode=2
Sep 17 18:29:22.197805 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:22.197647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerDied","Data":"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"}
Sep 17 18:29:22.836535 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:22.836493    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 18:29:26.502826 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.502802    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:29:26.532427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config\") pod \"b8949bcd-771a-4acf-914e-b827ba13728c\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") "
Sep 17 18:29:26.532584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqztt\" (UniqueName: \"kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt\") pod \"b8949bcd-771a-4acf-914e-b827ba13728c\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") "
Sep 17 18:29:26.532584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") pod \"b8949bcd-771a-4acf-914e-b827ba13728c\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") "
Sep 17 18:29:26.532584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location\") pod \"b8949bcd-771a-4acf-914e-b827ba13728c\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") "
Sep 17 18:29:26.532584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert\") pod \"b8949bcd-771a-4acf-914e-b827ba13728c\" (UID: \"b8949bcd-771a-4acf-914e-b827ba13728c\") "
Sep 17 18:29:26.532867 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8949bcd-771a-4acf-914e-b827ba13728c" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:26.532971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8949bcd-771a-4acf-914e-b827ba13728c" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:26.532971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.532963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8949bcd-771a-4acf-914e-b827ba13728c" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:26.534597 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.534578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8949bcd-771a-4acf-914e-b827ba13728c" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:26.534681 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.534619    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt" (OuterVolumeSpecName: "kube-api-access-tqztt") pod "b8949bcd-771a-4acf-914e-b827ba13728c" (UID: "b8949bcd-771a-4acf-914e-b827ba13728c"). InnerVolumeSpecName "kube-api-access-tqztt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:26.634111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.634083    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8949bcd-771a-4acf-914e-b827ba13728c-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:29:26.634111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.634109    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8949bcd-771a-4acf-914e-b827ba13728c-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:29:26.634111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.634118    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:29:26.634321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.634128    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8949bcd-771a-4acf-914e-b827ba13728c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:29:26.634321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:26.634136    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqztt\" (UniqueName: \"kubernetes.io/projected/b8949bcd-771a-4acf-914e-b827ba13728c-kube-api-access-tqztt\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:29:27.230430 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.230335    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8949bcd-771a-4acf-914e-b827ba13728c" containerID="868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593" exitCode=0
Sep 17 18:29:27.230430 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.230404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"
Sep 17 18:29:27.230430 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.230414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerDied","Data":"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"}
Sep 17 18:29:27.230662 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.230497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs" event={"ID":"b8949bcd-771a-4acf-914e-b827ba13728c","Type":"ContainerDied","Data":"06dfee3c84476ab84e0b5ce0fd83a866472d044b6d31405294a1191ca724f713"}
Sep 17 18:29:27.230662 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.230523    2577 scope.go:117] "RemoveContainer" containerID="a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"
Sep 17 18:29:27.239214 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.239199    2577 scope.go:117] "RemoveContainer" containerID="868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"
Sep 17 18:29:27.246840 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.246822    2577 scope.go:117] "RemoveContainer" containerID="0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"
Sep 17 18:29:27.248925 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.248901    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:29:27.253968 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.253946    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-5bqqs"]
Sep 17 18:29:27.254349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.254336    2577 scope.go:117] "RemoveContainer" containerID="a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"
Sep 17 18:29:27.254601 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:29:27.254586    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a\": container with ID starting with a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a not found: ID does not exist" containerID="a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"
Sep 17 18:29:27.254646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.254608    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a"} err="failed to get container status \"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a\": rpc error: code = NotFound desc = could not find container \"a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a\": container with ID starting with a43848b7597bfff25b1905bbe0ea45dff8695ca147f23493592d75058204445a not found: ID does not exist"
Sep 17 18:29:27.254646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.254624    2577 scope.go:117] "RemoveContainer" containerID="868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"
Sep 17 18:29:27.254887 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:29:27.254869    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593\": container with ID starting with 868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593 not found: ID does not exist" containerID="868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"
Sep 17 18:29:27.254962 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.254897    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593"} err="failed to get container status \"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593\": rpc error: code = NotFound desc = could not find container \"868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593\": container with ID starting with 868dc8cd961e71b0fe5df04d485b67aaf6b559c77b99944ac12f94a50890e593 not found: ID does not exist"
Sep 17 18:29:27.254962 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.254919    2577 scope.go:117] "RemoveContainer" containerID="0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"
Sep 17 18:29:27.255189 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:29:27.255170    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f\": container with ID starting with 0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f not found: ID does not exist" containerID="0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"
Sep 17 18:29:27.255231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:27.255193    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f"} err="failed to get container status \"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f\": rpc error: code = NotFound desc = could not find container \"0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f\": container with ID starting with 0ea68e715d2d5e9c885e0e5615a4b47c5df832495b87b764f1e392fc4f9b702f not found: ID does not exist"
Sep 17 18:29:28.942183 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:28.942146    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" path="/var/lib/kubelet/pods/b8949bcd-771a-4acf-914e-b827ba13728c/volumes"
Sep 17 18:29:30.018524 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:30.018486    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:30.193178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:30.193147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:40.018093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:40.018057    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:50.017925 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:50.017886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:29:57.937919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:29:57.937888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:30:06.098778 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.098720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:30:06.099246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.099078    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" containerID="cri-o://c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3" gracePeriod=30
Sep 17 18:30:06.099246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.099158    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kube-rbac-proxy" containerID="cri-o://58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c" gracePeriod=30
Sep 17 18:30:06.103363 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.103340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.227307 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.227240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.418546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.418511    2577 generic.go:356] "Generic (PLEG): container finished" podID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerID="58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c" exitCode=2
Sep 17 18:30:06.418739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:06.418571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerDied","Data":"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"}
Sep 17 18:30:07.937523 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:07.937483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:30:10.013586 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.013548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:30:10.744424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.744402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:30:10.810246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config\") pod \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") "
Sep 17 18:30:10.810434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location\") pod \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") "
Sep 17 18:30:10.810434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert\") pod \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") "
Sep 17 18:30:10.810434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") pod \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") "
Sep 17 18:30:10.810434 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp27p\" (UniqueName: \"kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p\") pod \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\" (UID: \"994d8bc3-4f09-4d4e-b765-49299ab7f05a\") "
Sep 17 18:30:10.810678 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "994d8bc3-4f09-4d4e-b765-49299ab7f05a" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:10.810678 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "994d8bc3-4f09-4d4e-b765-49299ab7f05a" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:10.810678 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.810659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "994d8bc3-4f09-4d4e-b765-49299ab7f05a" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:10.812355 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.812330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "994d8bc3-4f09-4d4e-b765-49299ab7f05a" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:10.812547 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.812526    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p" (OuterVolumeSpecName: "kube-api-access-kp27p") pod "994d8bc3-4f09-4d4e-b765-49299ab7f05a" (UID: "994d8bc3-4f09-4d4e-b765-49299ab7f05a"). InnerVolumeSpecName "kube-api-access-kp27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:10.911104 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.911080    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.911104 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.911102    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.911258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.911113    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994d8bc3-4f09-4d4e-b765-49299ab7f05a-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.911258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.911123    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994d8bc3-4f09-4d4e-b765-49299ab7f05a-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.911258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:10.911131    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp27p\" (UniqueName: \"kubernetes.io/projected/994d8bc3-4f09-4d4e-b765-49299ab7f05a-kube-api-access-kp27p\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:30:11.246293 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.246211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:11.443422 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.443384    2577 generic.go:356] "Generic (PLEG): container finished" podID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerID="c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3" exitCode=0
Sep 17 18:30:11.443587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.443437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"
Sep 17 18:30:11.443587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.443470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerDied","Data":"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"}
Sep 17 18:30:11.443587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.443535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw" event={"ID":"994d8bc3-4f09-4d4e-b765-49299ab7f05a","Type":"ContainerDied","Data":"ffe1251ec0a63f024aa19874a7e26fede082a6a06cee270a20bcf8468cf07286"}
Sep 17 18:30:11.443587 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.443563    2577 scope.go:117] "RemoveContainer" containerID="58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"
Sep 17 18:30:11.451966 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.451949    2577 scope.go:117] "RemoveContainer" containerID="c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"
Sep 17 18:30:11.459424 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.459394    2577 scope.go:117] "RemoveContainer" containerID="31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"
Sep 17 18:30:11.461852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.461829    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:30:11.465513 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.465493    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c94b475d5-kf2xw"]
Sep 17 18:30:11.467680 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.467657    2577 scope.go:117] "RemoveContainer" containerID="58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"
Sep 17 18:30:11.467985 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:30:11.467960    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c\": container with ID starting with 58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c not found: ID does not exist" containerID="58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"
Sep 17 18:30:11.468044 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.467992    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c"} err="failed to get container status \"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c\": rpc error: code = NotFound desc = could not find container \"58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c\": container with ID starting with 58fd92c6ae3d57a6e86e78c1afd39e34b82015b5525c12f1277eea58bf98506c not found: ID does not exist"
Sep 17 18:30:11.468044 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.468009    2577 scope.go:117] "RemoveContainer" containerID="c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"
Sep 17 18:30:11.468262 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:30:11.468247    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3\": container with ID starting with c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3 not found: ID does not exist" containerID="c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"
Sep 17 18:30:11.468313 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.468265    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3"} err="failed to get container status \"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3\": rpc error: code = NotFound desc = could not find container \"c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3\": container with ID starting with c878c8fbe30a839d5c997c48ee2969154dd88fc3dee241be2b26e0e8994d76b3 not found: ID does not exist"
Sep 17 18:30:11.468313 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.468278    2577 scope.go:117] "RemoveContainer" containerID="31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"
Sep 17 18:30:11.468539 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:30:11.468521    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff\": container with ID starting with 31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff not found: ID does not exist" containerID="31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"
Sep 17 18:30:11.468600 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.468556    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff"} err="failed to get container status \"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff\": rpc error: code = NotFound desc = could not find container \"31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff\": container with ID starting with 31f3556288700c5a6804fae869be3151e3e2e715c5ad77e758d6340b38e7beff not found: ID does not exist"
Sep 17 18:30:11.571858 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:11.571786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.942574 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:12.942535    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" path="/var/lib/kubelet/pods/994d8bc3-4f09-4d4e-b765-49299ab7f05a/volumes"
Sep 17 18:30:52.394010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:52.393972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.943043 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:53.943011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.525405 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.525374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.558034 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558004    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:30:57.558341 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558355    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kube-rbac-proxy"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558361    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kube-rbac-proxy"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="storage-initializer"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558376    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="storage-initializer"
Sep 17 18:30:57.558385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558386    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558391    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="storage-initializer"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558410    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="storage-initializer"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558421    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kube-rbac-proxy"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558426    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kube-rbac-proxy"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558475    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kube-rbac-proxy"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558484    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kserve-container"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558490    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="994d8bc3-4f09-4d4e-b765-49299ab7f05a" containerName="kserve-container"
Sep 17 18:30:57.558578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.558497    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8949bcd-771a-4acf-914e-b827ba13728c" containerName="kube-rbac-proxy"
Sep 17 18:30:57.561666 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.561651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.564763 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.564736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:30:57.564881 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.564783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 18:30:57.569910 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.569891    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:30:57.676235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.676205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.689261 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.689238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.689333 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.689294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlx9r\" (UniqueName: \"kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.689383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.689337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.689383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.689359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.689456 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.689417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790255 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790255 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlx9r\" (UniqueName: \"kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790460 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.790984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.790965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.792843 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.792825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.798218 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.798199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlx9r\" (UniqueName: \"kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mzv88\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:57.872298 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:57.872266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:30:58.141493 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:58.141466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:30:58.143420 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:30:58.143397    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f8f180_6370_45f3_ae3e_eec1e86e63fd.slice/crio-bf432d03490fddcb8954b294c5f9836cdffb317b5677ae3b588b29e0d4ebcd1f WatchSource:0}: Error finding container bf432d03490fddcb8954b294c5f9836cdffb317b5677ae3b588b29e0d4ebcd1f: Status 404 returned error can't find the container with id bf432d03490fddcb8954b294c5f9836cdffb317b5677ae3b588b29e0d4ebcd1f
Sep 17 18:30:58.650770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:58.650718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerStarted","Data":"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"}
Sep 17 18:30:58.651239 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:30:58.650793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerStarted","Data":"bf432d03490fddcb8954b294c5f9836cdffb317b5677ae3b588b29e0d4ebcd1f"}
Sep 17 18:31:00.335895 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.335854    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:31:00.336365 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.336183    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" containerID="cri-o://c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47" gracePeriod=30
Sep 17 18:31:00.336365 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.336255    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kube-rbac-proxy" containerID="cri-o://92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1" gracePeriod=30
Sep 17 18:31:00.358324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.358303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.661322 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.661294    2577 generic.go:356] "Generic (PLEG): container finished" podID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerID="92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1" exitCode=2
Sep 17 18:31:00.661469 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:00.661367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerDied","Data":"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"}
Sep 17 18:31:01.830745 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:01.830702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 18:31:01.836451 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:01.836424    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:31:03.677563 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:03.677529    2577 generic.go:356] "Generic (PLEG): container finished" podID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerID="da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7" exitCode=0
Sep 17 18:31:03.678094 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:03.677607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerDied","Data":"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"}
Sep 17 18:31:03.679033 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:03.679019    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:31:04.581585 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:04.581549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:05.396053 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.396026    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:31:05.452491 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452416    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location\") pod \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") "
Sep 17 18:31:05.452491 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert\") pod \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") "
Sep 17 18:31:05.452718 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls\") pod \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") "
Sep 17 18:31:05.452808 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" (UID: "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:05.452876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd2rc\" (UniqueName: \"kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc\") pod \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") "
Sep 17 18:31:05.452876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" (UID: "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:05.452999 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.452907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config\") pod \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\" (UID: \"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b\") "
Sep 17 18:31:05.453181 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.453162    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.453257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.453187    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.453514 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.453485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" (UID: "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:05.455148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.455122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" (UID: "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:05.455301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.455213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc" (OuterVolumeSpecName: "kube-api-access-zd2rc") pod "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" (UID: "74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b"). InnerVolumeSpecName "kube-api-access-zd2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:05.495163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.495120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:05.522717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.522693    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:31:05.523097 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523073    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="storage-initializer"
Sep 17 18:31:05.523097 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="storage-initializer"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523107    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523113    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523135    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kube-rbac-proxy"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523140    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kube-rbac-proxy"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kserve-container"
Sep 17 18:31:05.523235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.523200    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerName="kube-rbac-proxy"
Sep 17 18:31:05.531286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.531256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.533770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.533716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:31:05.533977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.533958    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:31:05.535108 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.535081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:31:05.554089 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.554199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.554199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.554199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.554199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.554414 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554346    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.554414 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554368    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2rc\" (UniqueName: \"kubernetes.io/projected/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-api-access-zd2rc\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.554414 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.554385    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.566833 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.566805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:05.604203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.604173    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:31:05.608121 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.608102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.610528 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.610506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:31:05.617828 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.617807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:31:05.655115 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.655247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655247 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.655250    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.655314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls podName:a0e01f8e-ed81-4248-8c66-9d12a3792409 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:06.155295024 +0000 UTC m=+2077.479616426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls") pod "isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.655833 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85wv\" (UniqueName: \"kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.655833 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.655922 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.655881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.656087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.656071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.669242 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.669215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:05.690283 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.690255    2577 generic.go:356] "Generic (PLEG): container finished" podID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" containerID="c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47" exitCode=0
Sep 17 18:31:05.690382 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.690339    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"
Sep 17 18:31:05.690382 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.690338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerDied","Data":"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"}
Sep 17 18:31:05.690509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.690409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq" event={"ID":"74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b","Type":"ContainerDied","Data":"f7ede3f4d80dd6093ae2678c0791f08d15d68f6732e70ad42efb6a0d91e1e42f"}
Sep 17 18:31:05.690509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.690452    2577 scope.go:117] "RemoveContainer" containerID="92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"
Sep 17 18:31:05.699919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.699795    2577 scope.go:117] "RemoveContainer" containerID="c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"
Sep 17 18:31:05.708771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.708728    2577 scope.go:117] "RemoveContainer" containerID="663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"
Sep 17 18:31:05.717398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.717378    2577 scope.go:117] "RemoveContainer" containerID="92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"
Sep 17 18:31:05.717712 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.717687    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1\": container with ID starting with 92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1 not found: ID does not exist" containerID="92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"
Sep 17 18:31:05.717811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.717718    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1"} err="failed to get container status \"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1\": rpc error: code = NotFound desc = could not find container \"92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1\": container with ID starting with 92b845c558def305ee36367e5ad85220297290a57e68db81500c5754c4a103d1 not found: ID does not exist"
Sep 17 18:31:05.717811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.717734    2577 scope.go:117] "RemoveContainer" containerID="c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"
Sep 17 18:31:05.718011 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.717994    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47\": container with ID starting with c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47 not found: ID does not exist" containerID="c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"
Sep 17 18:31:05.718053 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.718012    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47"} err="failed to get container status \"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47\": rpc error: code = NotFound desc = could not find container \"c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47\": container with ID starting with c7c37d8c4e10138b9f2336607c898f4921868c26320521e29b57e2917c35ea47 not found: ID does not exist"
Sep 17 18:31:05.718053 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.718027    2577 scope.go:117] "RemoveContainer" containerID="663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"
Sep 17 18:31:05.718319 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.718295    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508\": container with ID starting with 663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508 not found: ID does not exist" containerID="663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"
Sep 17 18:31:05.718414 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.718326    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508"} err="failed to get container status \"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508\": rpc error: code = NotFound desc = could not find container \"663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508\": container with ID starting with 663aa639f22f4aef5f962d551210fc41bbaeaef4e76cb6e35577aa4c9f271508 not found: ID does not exist"
Sep 17 18:31:05.724570 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.724548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:31:05.743586 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.743559    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-hrpwq"]
Sep 17 18:31:05.756569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.756668 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.756668 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.756668 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.756849 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85wv\" (UniqueName: \"kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.756908 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.756844    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:31:05.756962 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:05.756930    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls podName:7bc72791-9750-4b33-b68b-7a5de52de804 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:06.256908578 +0000 UTC m=+2077.581230001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls") pod "isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:31:05.757021 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.756992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.757286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.757271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.757428 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.757407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:05.765618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:05.765590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85wv\" (UniqueName: \"kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:06.146422 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.146388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:06.161150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.161121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:06.161281 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:06.161247    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:31:06.161327 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:31:06.161312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls podName:a0e01f8e-ed81-4248-8c66-9d12a3792409 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:07.161294496 +0000 UTC m=+2078.485615897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls") pod "isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:31:06.262036 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.262004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:06.264700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.264674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-pc8r9\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:06.336976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.336942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:06.519895 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.519810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:06.944159 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:06.944126    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b" path="/var/lib/kubelet/pods/74cc4bf1-46cd-4152-82be-b9a5d4ed5f2b/volumes"
Sep 17 18:31:07.171811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.171775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:07.174282 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.174258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:07.344640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.344613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:07.473627 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.473602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:31:07.477194 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:31:07.477156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc72791_9750_4b33_b68b_7a5de52de804.slice/crio-d17c4c993873f96716c4b9397000f46d09e0d80e16573b4c5cc15fd5e06ce53f WatchSource:0}: Error finding container d17c4c993873f96716c4b9397000f46d09e0d80e16573b4c5cc15fd5e06ce53f: Status 404 returned error can't find the container with id d17c4c993873f96716c4b9397000f46d09e0d80e16573b4c5cc15fd5e06ce53f
Sep 17 18:31:07.621140 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.621116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:31:07.622912 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:31:07.622885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e01f8e_ed81_4248_8c66_9d12a3792409.slice/crio-60da088d04f7ced59a9c1a15e393629dacfb5ce00c08adeba50b5d04ebd957b9 WatchSource:0}: Error finding container 60da088d04f7ced59a9c1a15e393629dacfb5ce00c08adeba50b5d04ebd957b9: Status 404 returned error can't find the container with id 60da088d04f7ced59a9c1a15e393629dacfb5ce00c08adeba50b5d04ebd957b9
Sep 17 18:31:07.704379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.704353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerStarted","Data":"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"}
Sep 17 18:31:07.704492 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.704420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerStarted","Data":"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"}
Sep 17 18:31:07.704650 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.704630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:31:07.706149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.706125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerStarted","Data":"37395e79600db5428bc6d790185b24fbb064d9c2ea1354db7e689730f35e682d"}
Sep 17 18:31:07.706274 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.706187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerStarted","Data":"60da088d04f7ced59a9c1a15e393629dacfb5ce00c08adeba50b5d04ebd957b9"}
Sep 17 18:31:07.707799 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.707778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerStarted","Data":"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"}
Sep 17 18:31:07.707903 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.707830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerStarted","Data":"d17c4c993873f96716c4b9397000f46d09e0d80e16573b4c5cc15fd5e06ce53f"}
Sep 17 18:31:07.724413 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:07.724327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podStartSLOduration=2.29204508 podStartE2EDuration="10.724312994s" totalImagesPullingTime="3.584071249s" totalInitContainerRuntime="4.848196665s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:07.723190604 +0000 UTC m=+2079.047512028" watchObservedRunningTime="2026-09-17 18:31:07.724312994 +0000 UTC m=+2079.048634417"
Sep 17 18:31:08.712720 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:08.712689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:31:08.714047 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:08.714022    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:31:09.717553 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:09.717509    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:31:11.728301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:11.728267    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerID="37395e79600db5428bc6d790185b24fbb064d9c2ea1354db7e689730f35e682d" exitCode=0
Sep 17 18:31:11.728721 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:11.728346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerDied","Data":"37395e79600db5428bc6d790185b24fbb064d9c2ea1354db7e689730f35e682d"}
Sep 17 18:31:11.729854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:11.729834    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bc72791-9750-4b33-b68b-7a5de52de804" containerID="b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1" exitCode=0
Sep 17 18:31:11.729944 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:11.729869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerDied","Data":"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"}
Sep 17 18:31:12.736860 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.736825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerStarted","Data":"be642e5f4f7e2d07197351a962f4b5e333202a5dd48ecd716db63786240ca13e"}
Sep 17 18:31:12.737248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.736901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerStarted","Data":"be8232fc370ea5ce0fd8e33550c1e38992449662396336e0e6f9aebad46725e3"}
Sep 17 18:31:12.737248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.737202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:12.737366 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.737263    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:12.738939 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.738914    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:12.739166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.739141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerStarted","Data":"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"}
Sep 17 18:31:12.739223 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.739194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerStarted","Data":"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"}
Sep 17 18:31:12.739423 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.739409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:12.739467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.739451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:12.740470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.740448    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:12.759065 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.759018    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podStartSLOduration=3.89289819 podStartE2EDuration="7.759003029s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866104839s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:12.756557606 +0000 UTC m=+2084.080879096" watchObservedRunningTime="2026-09-17 18:31:12.759003029 +0000 UTC m=+2084.083324453"
Sep 17 18:31:12.777508 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:12.777456    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podStartSLOduration=3.778341386 podStartE2EDuration="7.777440315s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.999098929s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:12.776212403 +0000 UTC m=+2084.100533827" watchObservedRunningTime="2026-09-17 18:31:12.777440315 +0000 UTC m=+2084.101761739"
Sep 17 18:31:13.750152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:13.750107    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:13.750574 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:13.750116    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:14.722490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:14.722457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:31:14.722952 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:14.722928    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:31:18.755209 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:18.755181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:31:18.755710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:18.755694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:31:18.755804 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:18.755776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:18.756140 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:18.756116    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:24.723219 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:24.723187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:31:28.756270 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:28.756222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:28.756668 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:28.756230    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:38.756620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:38.756578    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:38.757020 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:38.756578    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:39.100265 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.100179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:31:39.100477 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.100452    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container" containerID="cri-o://65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622" gracePeriod=30
Sep 17 18:31:39.100560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.100486    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" containerID="cri-o://88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e" gracePeriod=30
Sep 17 18:31:39.124015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.123972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:39.718731 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.718690    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:31:39.879877 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.879848    2577 generic.go:356] "Generic (PLEG): container finished" podID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerID="88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e" exitCode=2
Sep 17 18:31:39.880257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:39.879917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerDied","Data":"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"}
Sep 17 18:31:44.264541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:44.264513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:44.382338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:44.382302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:44.718037 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:44.717998    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:31:46.925212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:46.925176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:48.756671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:48.756632    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:48.756671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:48.756656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:49.717659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:49.717617    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:31:49.717988 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:49.717739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:31:54.718307 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:54.718261    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:31:58.755698 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:58.755655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:31:58.756207 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:58.756132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:31:59.718578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:31:59.718536    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:32:04.718598 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:04.718559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:32:08.755878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:08.755840    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:32:08.756285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:08.756054    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:32:09.735679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.735653    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:32:09.821525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.821477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls\") pod \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") "
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.821550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlx9r\" (UniqueName: \"kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r\") pod \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") "
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.821591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config\") pod \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") "
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.821650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location\") pod \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") "
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.821692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert\") pod \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\" (UID: \"78f8f180-6370-45f3-ae3e-eec1e86e63fd\") "
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.822058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78f8f180-6370-45f3-ae3e-eec1e86e63fd" (UID: "78f8f180-6370-45f3-ae3e-eec1e86e63fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:09.822068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.822063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78f8f180-6370-45f3-ae3e-eec1e86e63fd" (UID: "78f8f180-6370-45f3-ae3e-eec1e86e63fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:09.823696 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.823673    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78f8f180-6370-45f3-ae3e-eec1e86e63fd" (UID: "78f8f180-6370-45f3-ae3e-eec1e86e63fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:09.823996 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.823978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r" (OuterVolumeSpecName: "kube-api-access-jlx9r") pod "78f8f180-6370-45f3-ae3e-eec1e86e63fd" (UID: "78f8f180-6370-45f3-ae3e-eec1e86e63fd"). InnerVolumeSpecName "kube-api-access-jlx9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:09.833654 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.833626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78f8f180-6370-45f3-ae3e-eec1e86e63fd" (UID: "78f8f180-6370-45f3-ae3e-eec1e86e63fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:09.923080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.922984    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlx9r\" (UniqueName: \"kubernetes.io/projected/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-api-access-jlx9r\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:09.923080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.923026    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:09.923080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.923037    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f8f180-6370-45f3-ae3e-eec1e86e63fd-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:09.923080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.923048    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78f8f180-6370-45f3-ae3e-eec1e86e63fd-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:09.923080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:09.923057    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78f8f180-6370-45f3-ae3e-eec1e86e63fd-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.027593 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.027563    2577 generic.go:356] "Generic (PLEG): container finished" podID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerID="65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622" exitCode=137
Sep 17 18:32:10.027764 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.027610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerDied","Data":"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"}
Sep 17 18:32:10.027764 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.027658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" event={"ID":"78f8f180-6370-45f3-ae3e-eec1e86e63fd","Type":"ContainerDied","Data":"bf432d03490fddcb8954b294c5f9836cdffb317b5677ae3b588b29e0d4ebcd1f"}
Sep 17 18:32:10.027764 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.027613    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"
Sep 17 18:32:10.027764 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.027690    2577 scope.go:117] "RemoveContainer" containerID="88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"
Sep 17 18:32:10.039667 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.039649    2577 scope.go:117] "RemoveContainer" containerID="65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"
Sep 17 18:32:10.047384 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.047367    2577 scope.go:117] "RemoveContainer" containerID="da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"
Sep 17 18:32:10.052671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.052651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:32:10.055313 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.055294    2577 scope.go:117] "RemoveContainer" containerID="88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"
Sep 17 18:32:10.055588 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:32:10.055569    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e\": container with ID starting with 88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e not found: ID does not exist" containerID="88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"
Sep 17 18:32:10.055643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.055597    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e"} err="failed to get container status \"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e\": rpc error: code = NotFound desc = could not find container \"88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e\": container with ID starting with 88daf52e7e26dd9c80dc3b5264c4b7e905e56d7fe0363d71628d0ced84fa6a8e not found: ID does not exist"
Sep 17 18:32:10.055643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.055612    2577 scope.go:117] "RemoveContainer" containerID="65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"
Sep 17 18:32:10.055930 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:32:10.055906    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622\": container with ID starting with 65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622 not found: ID does not exist" containerID="65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"
Sep 17 18:32:10.055995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.055936    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622"} err="failed to get container status \"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622\": rpc error: code = NotFound desc = could not find container \"65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622\": container with ID starting with 65be96117abf5d994ba3e9e7aba167c6f7700ed6c8ee9b3c73a1b8ab0552e622 not found: ID does not exist"
Sep 17 18:32:10.055995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.055956    2577 scope.go:117] "RemoveContainer" containerID="da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"
Sep 17 18:32:10.056228 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:32:10.056204    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7\": container with ID starting with da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7 not found: ID does not exist" containerID="da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"
Sep 17 18:32:10.056273 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.056236    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7"} err="failed to get container status \"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7\": rpc error: code = NotFound desc = could not find container \"da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7\": container with ID starting with da594addca0a56d1da0034531d4d97d4d769dd0ef5e8ce9e00f25d15ec84dea7 not found: ID does not exist"
Sep 17 18:32:10.056786 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.056770    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88"]
Sep 17 18:32:10.718010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.717974    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mzv88" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": context deadline exceeded"
Sep 17 18:32:10.941953 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:10.941917    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" path="/var/lib/kubelet/pods/78f8f180-6370-45f3-ae3e-eec1e86e63fd/volumes"
Sep 17 18:32:18.756170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:18.756125    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:32:18.756604 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:18.756126    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:32:28.756953 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:28.756911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:32:28.757490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:28.757340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:32:35.552153 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:35.552117    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:32:35.552546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:35.552346    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" containerID="cri-o://be8232fc370ea5ce0fd8e33550c1e38992449662396336e0e6f9aebad46725e3" gracePeriod=30
Sep 17 18:32:35.552546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:35.552392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kube-rbac-proxy" containerID="cri-o://be642e5f4f7e2d07197351a962f4b5e333202a5dd48ecd716db63786240ca13e" gracePeriod=30
Sep 17 18:32:35.577354 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:35.577324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:36.156263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:36.156231    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerID="be642e5f4f7e2d07197351a962f4b5e333202a5dd48ecd716db63786240ca13e" exitCode=2
Sep 17 18:32:36.156435 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:36.156267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerDied","Data":"be642e5f4f7e2d07197351a962f4b5e333202a5dd48ecd716db63786240ca13e"}
Sep 17 18:32:38.751101 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:38.751059    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:32:38.756260 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:38.756220    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:32:40.178006 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.177966    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerID="be8232fc370ea5ce0fd8e33550c1e38992449662396336e0e6f9aebad46725e3" exitCode=0
Sep 17 18:32:40.178006 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.178006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerDied","Data":"be8232fc370ea5ce0fd8e33550c1e38992449662396336e0e6f9aebad46725e3"}
Sep 17 18:32:40.201823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.201797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:32:40.282976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.282884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") pod \"a0e01f8e-ed81-4248-8c66-9d12a3792409\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") "
Sep 17 18:32:40.282976 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.282955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location\") pod \"a0e01f8e-ed81-4248-8c66-9d12a3792409\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") "
Sep 17 18:32:40.283228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283008    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert\") pod \"a0e01f8e-ed81-4248-8c66-9d12a3792409\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") "
Sep 17 18:32:40.283228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config\") pod \"a0e01f8e-ed81-4248-8c66-9d12a3792409\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") "
Sep 17 18:32:40.283228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7\") pod \"a0e01f8e-ed81-4248-8c66-9d12a3792409\" (UID: \"a0e01f8e-ed81-4248-8c66-9d12a3792409\") "
Sep 17 18:32:40.283374 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0e01f8e-ed81-4248-8c66-9d12a3792409" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:40.283374 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0e01f8e-ed81-4248-8c66-9d12a3792409" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:40.283448 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.283378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0e01f8e-ed81-4248-8c66-9d12a3792409" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:40.285067 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.285043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0e01f8e-ed81-4248-8c66-9d12a3792409" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:40.285212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.285191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7" (OuterVolumeSpecName: "kube-api-access-sdkp7") pod "a0e01f8e-ed81-4248-8c66-9d12a3792409" (UID: "a0e01f8e-ed81-4248-8c66-9d12a3792409"). InnerVolumeSpecName "kube-api-access-sdkp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:40.384497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.384463    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e01f8e-ed81-4248-8c66-9d12a3792409-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:40.384497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.384496    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:40.384700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.384510    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:40.384700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.384544    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/a0e01f8e-ed81-4248-8c66-9d12a3792409-kube-api-access-sdkp7\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:40.384700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:40.384554    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e01f8e-ed81-4248-8c66-9d12a3792409-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.184912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.184880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs" event={"ID":"a0e01f8e-ed81-4248-8c66-9d12a3792409","Type":"ContainerDied","Data":"60da088d04f7ced59a9c1a15e393629dacfb5ce00c08adeba50b5d04ebd957b9"}
Sep 17 18:32:41.184912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.184907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"
Sep 17 18:32:41.185367 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.184953    2577 scope.go:117] "RemoveContainer" containerID="be642e5f4f7e2d07197351a962f4b5e333202a5dd48ecd716db63786240ca13e"
Sep 17 18:32:41.193648 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.193627    2577 scope.go:117] "RemoveContainer" containerID="be8232fc370ea5ce0fd8e33550c1e38992449662396336e0e6f9aebad46725e3"
Sep 17 18:32:41.201394 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.201344    2577 scope.go:117] "RemoveContainer" containerID="37395e79600db5428bc6d790185b24fbb064d9c2ea1354db7e689730f35e682d"
Sep 17 18:32:41.203648 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.203624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:32:41.207150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:41.207127    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-jw9bs"]
Sep 17 18:32:42.941877 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:32:42.941846    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" path="/var/lib/kubelet/pods/a0e01f8e-ed81-4248-8c66-9d12a3792409/volumes"
Sep 17 18:33:56.783739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:33:56.783702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.916212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:34:01.916174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:02.666767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:34:02.666715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:15.745315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.745242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:35:15.745826 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.745503    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" containerID="cri-o://f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1" gracePeriod=30
Sep 17 18:35:15.745826 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.745528    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kube-rbac-proxy" containerID="cri-o://45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725" gracePeriod=30
Sep 17 18:35:15.779225 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.779203    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:15.864470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.864444    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bc72791-9750-4b33-b68b-7a5de52de804" containerID="45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725" exitCode=2
Sep 17 18:35:15.864589 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:15.864516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerDied","Data":"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"}
Sep 17 18:35:18.750713 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:18.750669    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 18:35:18.755933 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:18.755908    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:35:20.184415 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.184390    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:35:20.248425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert\") pod \"7bc72791-9750-4b33-b68b-7a5de52de804\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") "
Sep 17 18:35:20.248584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248444    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location\") pod \"7bc72791-9750-4b33-b68b-7a5de52de804\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") "
Sep 17 18:35:20.248584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config\") pod \"7bc72791-9750-4b33-b68b-7a5de52de804\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") "
Sep 17 18:35:20.248584 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248524    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n85wv\" (UniqueName: \"kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv\") pod \"7bc72791-9750-4b33-b68b-7a5de52de804\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") "
Sep 17 18:35:20.248774 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") pod \"7bc72791-9750-4b33-b68b-7a5de52de804\" (UID: \"7bc72791-9750-4b33-b68b-7a5de52de804\") "
Sep 17 18:35:20.248839 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bc72791-9750-4b33-b68b-7a5de52de804" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:20.248839 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bc72791-9750-4b33-b68b-7a5de52de804" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:20.248944 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bc72791-9750-4b33-b68b-7a5de52de804" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:20.248944 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248929    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.248944 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.248943    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc72791-9750-4b33-b68b-7a5de52de804-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.250732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.250704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv" (OuterVolumeSpecName: "kube-api-access-n85wv") pod "7bc72791-9750-4b33-b68b-7a5de52de804" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804"). InnerVolumeSpecName "kube-api-access-n85wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:20.250732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.250705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bc72791-9750-4b33-b68b-7a5de52de804" (UID: "7bc72791-9750-4b33-b68b-7a5de52de804"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:20.349582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.349553    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc72791-9750-4b33-b68b-7a5de52de804-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.349582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.349576    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85wv\" (UniqueName: \"kubernetes.io/projected/7bc72791-9750-4b33-b68b-7a5de52de804-kube-api-access-n85wv\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.349582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.349586    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc72791-9750-4b33-b68b-7a5de52de804-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.889949 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.889911    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bc72791-9750-4b33-b68b-7a5de52de804" containerID="f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1" exitCode=0
Sep 17 18:35:20.890116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.889978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"
Sep 17 18:35:20.890116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.889987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerDied","Data":"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"}
Sep 17 18:35:20.890116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.890058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9" event={"ID":"7bc72791-9750-4b33-b68b-7a5de52de804","Type":"ContainerDied","Data":"d17c4c993873f96716c4b9397000f46d09e0d80e16573b4c5cc15fd5e06ce53f"}
Sep 17 18:35:20.890116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.890098    2577 scope.go:117] "RemoveContainer" containerID="45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"
Sep 17 18:35:20.900775 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.900741    2577 scope.go:117] "RemoveContainer" containerID="f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"
Sep 17 18:35:20.908850 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.908826    2577 scope.go:117] "RemoveContainer" containerID="b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"
Sep 17 18:35:20.917248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.917225    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:35:20.917863 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.917846    2577 scope.go:117] "RemoveContainer" containerID="45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"
Sep 17 18:35:20.918132 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:35:20.918114    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725\": container with ID starting with 45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725 not found: ID does not exist" containerID="45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"
Sep 17 18:35:20.918196 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.918148    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725"} err="failed to get container status \"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725\": rpc error: code = NotFound desc = could not find container \"45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725\": container with ID starting with 45991aa2f99f7000cb8de31156937425bcc85af759fc727c509096c089652725 not found: ID does not exist"
Sep 17 18:35:20.918196 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.918178    2577 scope.go:117] "RemoveContainer" containerID="f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"
Sep 17 18:35:20.918455 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:35:20.918434    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1\": container with ID starting with f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1 not found: ID does not exist" containerID="f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"
Sep 17 18:35:20.918504 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.918460    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1"} err="failed to get container status \"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1\": rpc error: code = NotFound desc = could not find container \"f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1\": container with ID starting with f5a6d2c3a12f776546569d7b33846f1ac1835b07a4243fe9a0f4c2c0cf62a4b1 not found: ID does not exist"
Sep 17 18:35:20.918504 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.918476    2577 scope.go:117] "RemoveContainer" containerID="b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"
Sep 17 18:35:20.918732 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:35:20.918714    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1\": container with ID starting with b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1 not found: ID does not exist" containerID="b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"
Sep 17 18:35:20.918815 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.918770    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1"} err="failed to get container status \"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1\": rpc error: code = NotFound desc = could not find container \"b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1\": container with ID starting with b6802642899c325a2073740eb0379aa855359682d47fe53cde717e4ad61602c1 not found: ID does not exist"
Sep 17 18:35:20.926267 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.926239    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-pc8r9"]
Sep 17 18:35:20.939136 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.939078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:20.942687 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.942663    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" path="/var/lib/kubelet/pods/7bc72791-9750-4b33-b68b-7a5de52de804/volumes"
Sep 17 18:35:20.979178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:35:20.979463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979452    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979465    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979479    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979487    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979492    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979508    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container"
Sep 17 18:35:20.979521 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979516    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979525    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979531    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979536    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979541    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979554    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979561    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979566    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979576    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="storage-initializer"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kserve-container"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979640    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979647    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kserve-container"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc72791-9750-4b33-b68b-7a5de52de804" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979658    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f8f180-6370-45f3-ae3e-eec1e86e63fd" containerName="kube-rbac-proxy"
Sep 17 18:35:20.979741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.979668    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e01f8e-ed81-4248-8c66-9d12a3792409" containerName="kserve-container"
Sep 17 18:35:20.983876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.983860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:20.986550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986522    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:35:20.986550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:35:20.986550 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:35:20.986878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:35:20.986878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:35:20.986878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986821    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:35:20.986878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.986869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:35:20.994484 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:20.994465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:35:21.054663 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.054632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.054824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.054689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.054824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.054744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.054824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.054787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.054824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.054811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv67n\" (UniqueName: \"kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.066627 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.066601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:21.155467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.155467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.155467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.155732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv67n\" (UniqueName: \"kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.155732 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.155881 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.155823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.156207 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.156188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.156257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.156204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.157902 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.157887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.163000 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.162978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv67n\" (UniqueName: \"kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n\") pod \"isvc-canary-stop-predictor-b84cdfc7-rbmlc\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.182320 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.182290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:21.294858 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.294827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:21.298148 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.298083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:21.568952 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.568927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:35:21.571283 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:35:21.571250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823643ac_34ed_4e6f_a35a_dcdac8286267.slice/crio-e85501ab10c2b47db5d4601a6be452e78f56cf942c77bc916dd8a7eb986f63df WatchSource:0}: Error finding container e85501ab10c2b47db5d4601a6be452e78f56cf942c77bc916dd8a7eb986f63df: Status 404 returned error can't find the container with id e85501ab10c2b47db5d4601a6be452e78f56cf942c77bc916dd8a7eb986f63df
Sep 17 18:35:21.897518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.897485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerStarted","Data":"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"}
Sep 17 18:35:21.897691 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:21.897547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerStarted","Data":"e85501ab10c2b47db5d4601a6be452e78f56cf942c77bc916dd8a7eb986f63df"}
Sep 17 18:35:25.917140 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:25.917102    2577 generic.go:356] "Generic (PLEG): container finished" podID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerID="aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56" exitCode=0
Sep 17 18:35:25.917538 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:25.917150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerDied","Data":"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"}
Sep 17 18:35:26.923479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.923441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerStarted","Data":"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"}
Sep 17 18:35:26.923892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.923501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerStarted","Data":"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"}
Sep 17 18:35:26.923892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.923774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:26.924000 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.923915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:26.925079 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.925051    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:35:26.944588 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:26.944548    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podStartSLOduration=3.195212528 podStartE2EDuration="6.944534228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7493217s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:26.942823943 +0000 UTC m=+2338.267145366" watchObservedRunningTime="2026-09-17 18:35:26.944534228 +0000 UTC m=+2338.268855670"
Sep 17 18:35:27.931268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:27.931233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:35:32.935493 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:32.935467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:35:32.935990 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:32.935966    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:35:42.935918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:42.935877    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:35:52.575773 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:52.575727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:52.936435 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:52.936398    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:35:57.719619 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.719587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:57.750838 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.750804    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:35:57.754722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.754705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.756945 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.756925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:35:57.757055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.756974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:35:57.764446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.764424    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:35:57.781527 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.781501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.781650 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.781541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.781650 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.781574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.781744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.781648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggg9\" (UniqueName: \"kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.781815 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.781791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.882987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggg9\" (UniqueName: \"kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883186 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883186 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883418 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:35:57.883310    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:35:57.883418 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:35:57.883402    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls podName:9496fd41-592c-4813-8df1-549ff1216fd8 nodeName:}" failed. No retries permitted until 2026-09-17 18:35:58.383373822 +0000 UTC m=+2369.707695244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:35:57.883611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883678 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.883832 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.883816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:57.892256 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:57.892233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggg9\" (UniqueName: \"kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:58.389087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:58.389055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:58.391346 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:58.391321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:58.665009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:58.664919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:35:58.938722 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:35:58.938689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9496fd41_592c_4813_8df1_549ff1216fd8.slice/crio-6f23aaca11f4a6480650fcbb7afabc05b73f65959ed0a55d082b67815c71a000 WatchSource:0}: Error finding container 6f23aaca11f4a6480650fcbb7afabc05b73f65959ed0a55d082b67815c71a000: Status 404 returned error can't find the container with id 6f23aaca11f4a6480650fcbb7afabc05b73f65959ed0a55d082b67815c71a000
Sep 17 18:35:58.942725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:58.942705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:35:59.067314 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:59.067280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerStarted","Data":"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"}
Sep 17 18:35:59.067490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:59.067342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerStarted","Data":"6f23aaca11f4a6480650fcbb7afabc05b73f65959ed0a55d082b67815c71a000"}
Sep 17 18:35:59.695009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:35:59.694976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:02.936228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:02.936194    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:03.090574 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:03.090543    2577 generic.go:356] "Generic (PLEG): container finished" podID="9496fd41-592c-4813-8df1-549ff1216fd8" containerID="3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba" exitCode=0
Sep 17 18:36:03.090785 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:03.090589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerDied","Data":"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"}
Sep 17 18:36:04.098087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:04.098052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerStarted","Data":"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"}
Sep 17 18:36:04.098519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:04.098126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerStarted","Data":"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"}
Sep 17 18:36:04.098519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:04.098410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:04.121007 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:04.120957    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" podStartSLOduration=3.197126465 podStartE2EDuration="7.120941718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.923815253s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:04.119301258 +0000 UTC m=+2375.443622681" watchObservedRunningTime="2026-09-17 18:36:04.120941718 +0000 UTC m=+2375.445263141"
Sep 17 18:36:05.102501 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:05.102473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:11.112485 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:11.112457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:12.936682 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:12.936641    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:22.935915 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:22.935878    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:32.936373 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:32.936328    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:36.937164 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:36.937119    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:41.116893 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:41.116863    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:46.941688 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:46.941662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:36:47.798622 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:47.798590    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:36:47.798900 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:47.798877    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kserve-container" containerID="cri-o://43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79" gracePeriod=30
Sep 17 18:36:47.798984 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:47.798909    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kube-rbac-proxy" containerID="cri-o://b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1" gracePeriod=30
Sep 17 18:36:47.821234 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:47.821198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:48.295236 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:48.295202    2577 generic.go:356] "Generic (PLEG): container finished" podID="9496fd41-592c-4813-8df1-549ff1216fd8" containerID="b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1" exitCode=2
Sep 17 18:36:48.295620 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:48.295255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerDied","Data":"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"}
Sep 17 18:36:51.108684 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:51.108646    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 18:36:51.113012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:51.112989    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:36:52.492464 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.492427    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:36:52.492979 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.492727    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" containerID="cri-o://889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3" gracePeriod=30
Sep 17 18:36:52.492979 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.492746    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kube-rbac-proxy" containerID="cri-o://e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c" gracePeriod=30
Sep 17 18:36:52.931655 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.931616    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 18:36:52.949800 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.949765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:52.991478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.991440    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:36:52.995126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.995103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:52.997450 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.997427    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:36:52.997567 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:52.997515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:36:53.004675 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.004653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:36:53.056336 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.056298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.056336 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.056344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.056558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.056367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.056558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.056428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.056558 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.056477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2rw\" (UniqueName: \"kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.156898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.156855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.156898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.156904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.156926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.156956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157132 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.156995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2rw\" (UniqueName: \"kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157271 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:53.157126    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:36:53.157271 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:53.157204    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls podName:8bf46ca6-5485-44dc-931e-1f4f74cd6d70 nodeName:}" failed. No retries permitted until 2026-09-17 18:36:53.657187396 +0000 UTC m=+2424.981508803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:36:53.157367 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.157345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157607 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.157590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.157644 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.157624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.165320 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.165297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2rw\" (UniqueName: \"kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.319601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.319523    2577 generic.go:356] "Generic (PLEG): container finished" podID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerID="e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c" exitCode=2
Sep 17 18:36:53.319748 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.319595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerDied","Data":"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"}
Sep 17 18:36:53.478953 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.478913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:53.661130 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.661093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.663481 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.663463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:53.906375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:53.906340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:54.185010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.184983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:36:54.188230 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:36:54.188202    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf46ca6_5485_44dc_931e_1f4f74cd6d70.slice/crio-341a5d29909dec5b4c7e9e09272ab2c1075b05ee7a98bfa5f513e18add167b31 WatchSource:0}: Error finding container 341a5d29909dec5b4c7e9e09272ab2c1075b05ee7a98bfa5f513e18add167b31: Status 404 returned error can't find the container with id 341a5d29909dec5b4c7e9e09272ab2c1075b05ee7a98bfa5f513e18add167b31
Sep 17 18:36:54.231284 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.230097    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:54.265076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config\") pod \"9496fd41-592c-4813-8df1-549ff1216fd8\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") "
Sep 17 18:36:54.265263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") pod \"9496fd41-592c-4813-8df1-549ff1216fd8\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") "
Sep 17 18:36:54.265263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265121    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location\") pod \"9496fd41-592c-4813-8df1-549ff1216fd8\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") "
Sep 17 18:36:54.265263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zggg9\" (UniqueName: \"kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9\") pod \"9496fd41-592c-4813-8df1-549ff1216fd8\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") "
Sep 17 18:36:54.265263 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert\") pod \"9496fd41-592c-4813-8df1-549ff1216fd8\" (UID: \"9496fd41-592c-4813-8df1-549ff1216fd8\") "
Sep 17 18:36:54.265525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9496fd41-592c-4813-8df1-549ff1216fd8" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:54.265525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265475    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:54.265525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9496fd41-592c-4813-8df1-549ff1216fd8" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:54.265700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.265633    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9496fd41-592c-4813-8df1-549ff1216fd8" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:54.267331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.267308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9" (OuterVolumeSpecName: "kube-api-access-zggg9") pod "9496fd41-592c-4813-8df1-549ff1216fd8" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8"). InnerVolumeSpecName "kube-api-access-zggg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:54.267618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.267599    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9496fd41-592c-4813-8df1-549ff1216fd8" (UID: "9496fd41-592c-4813-8df1-549ff1216fd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:54.326731 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.326692    2577 generic.go:356] "Generic (PLEG): container finished" podID="9496fd41-592c-4813-8df1-549ff1216fd8" containerID="43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79" exitCode=0
Sep 17 18:36:54.326936 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.326789    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"
Sep 17 18:36:54.326936 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.326794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerDied","Data":"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"}
Sep 17 18:36:54.326936 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.326907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp" event={"ID":"9496fd41-592c-4813-8df1-549ff1216fd8","Type":"ContainerDied","Data":"6f23aaca11f4a6480650fcbb7afabc05b73f65959ed0a55d082b67815c71a000"}
Sep 17 18:36:54.327156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.326946    2577 scope.go:117] "RemoveContainer" containerID="b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"
Sep 17 18:36:54.328602 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.328581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerStarted","Data":"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"}
Sep 17 18:36:54.328769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.328739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerStarted","Data":"341a5d29909dec5b4c7e9e09272ab2c1075b05ee7a98bfa5f513e18add167b31"}
Sep 17 18:36:54.335823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.335805    2577 scope.go:117] "RemoveContainer" containerID="43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"
Sep 17 18:36:54.344003 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.343984    2577 scope.go:117] "RemoveContainer" containerID="3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"
Sep 17 18:36:54.352377 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.352354    2577 scope.go:117] "RemoveContainer" containerID="b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"
Sep 17 18:36:54.352696 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:54.352673    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1\": container with ID starting with b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1 not found: ID does not exist" containerID="b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"
Sep 17 18:36:54.352787 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.352713    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1"} err="failed to get container status \"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1\": rpc error: code = NotFound desc = could not find container \"b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1\": container with ID starting with b2288af805b810226ba5641731e58a7bea62d1ec5573eccbf74bd51625dfb6b1 not found: ID does not exist"
Sep 17 18:36:54.352787 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.352730    2577 scope.go:117] "RemoveContainer" containerID="43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"
Sep 17 18:36:54.353027 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:54.353009    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79\": container with ID starting with 43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79 not found: ID does not exist" containerID="43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"
Sep 17 18:36:54.353076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.353033    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79"} err="failed to get container status \"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79\": rpc error: code = NotFound desc = could not find container \"43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79\": container with ID starting with 43110633bfbb5a78316897452aec96256539904bf2711a3c180591460b052a79 not found: ID does not exist"
Sep 17 18:36:54.353076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.353049    2577 scope.go:117] "RemoveContainer" containerID="3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"
Sep 17 18:36:54.353330 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:54.353314    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba\": container with ID starting with 3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba not found: ID does not exist" containerID="3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"
Sep 17 18:36:54.353371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.353335    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba"} err="failed to get container status \"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba\": rpc error: code = NotFound desc = could not find container \"3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba\": container with ID starting with 3d49c2f1be7bb034e26167f6e739d6026555518549f7431a74e1c757c2e90eba not found: ID does not exist"
Sep 17 18:36:54.363340 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.363311    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:36:54.365472 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.365447    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-52fcp"]
Sep 17 18:36:54.366142 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.366120    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9496fd41-592c-4813-8df1-549ff1216fd8-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:54.366228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.366144    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zggg9\" (UniqueName: \"kubernetes.io/projected/9496fd41-592c-4813-8df1-549ff1216fd8-kube-api-access-zggg9\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:54.366228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.366155    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9496fd41-592c-4813-8df1-549ff1216fd8-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:54.366228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.366164    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9496fd41-592c-4813-8df1-549ff1216fd8-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:54.942599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:54.942513    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" path="/var/lib/kubelet/pods/9496fd41-592c-4813-8df1-549ff1216fd8/volumes"
Sep 17 18:36:55.950258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:55.950228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:56.012510 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:56.012472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:56.937179 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:56.937137    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:36:57.242738 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.242712    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:36:57.287670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.287636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls\") pod \"823643ac-34ed-4e6f-a35a-dcdac8286267\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") "
Sep 17 18:36:57.287670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.287677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv67n\" (UniqueName: \"kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n\") pod \"823643ac-34ed-4e6f-a35a-dcdac8286267\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") "
Sep 17 18:36:57.287929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.287725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config\") pod \"823643ac-34ed-4e6f-a35a-dcdac8286267\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") "
Sep 17 18:36:57.287929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.287743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location\") pod \"823643ac-34ed-4e6f-a35a-dcdac8286267\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") "
Sep 17 18:36:57.287929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.287895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert\") pod \"823643ac-34ed-4e6f-a35a-dcdac8286267\" (UID: \"823643ac-34ed-4e6f-a35a-dcdac8286267\") "
Sep 17 18:36:57.288105 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.288050    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "823643ac-34ed-4e6f-a35a-dcdac8286267" (UID: "823643ac-34ed-4e6f-a35a-dcdac8286267"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:57.288158 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.288095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "823643ac-34ed-4e6f-a35a-dcdac8286267" (UID: "823643ac-34ed-4e6f-a35a-dcdac8286267"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:57.288312 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.288290    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:57.288398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.288310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "823643ac-34ed-4e6f-a35a-dcdac8286267" (UID: "823643ac-34ed-4e6f-a35a-dcdac8286267"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:57.288398 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.288317    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823643ac-34ed-4e6f-a35a-dcdac8286267-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:57.289875 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.289853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "823643ac-34ed-4e6f-a35a-dcdac8286267" (UID: "823643ac-34ed-4e6f-a35a-dcdac8286267"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:57.289968 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.289923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n" (OuterVolumeSpecName: "kube-api-access-bv67n") pod "823643ac-34ed-4e6f-a35a-dcdac8286267" (UID: "823643ac-34ed-4e6f-a35a-dcdac8286267"). InnerVolumeSpecName "kube-api-access-bv67n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:57.347010 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.346971    2577 generic.go:356] "Generic (PLEG): container finished" podID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerID="889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3" exitCode=0
Sep 17 18:36:57.347156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.347020    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"
Sep 17 18:36:57.347156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.347052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerDied","Data":"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"}
Sep 17 18:36:57.347156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.347115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc" event={"ID":"823643ac-34ed-4e6f-a35a-dcdac8286267","Type":"ContainerDied","Data":"e85501ab10c2b47db5d4601a6be452e78f56cf942c77bc916dd8a7eb986f63df"}
Sep 17 18:36:57.347156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.347143    2577 scope.go:117] "RemoveContainer" containerID="e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"
Sep 17 18:36:57.356625 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.356607    2577 scope.go:117] "RemoveContainer" containerID="889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"
Sep 17 18:36:57.364192 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.364172    2577 scope.go:117] "RemoveContainer" containerID="aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"
Sep 17 18:36:57.370021 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.369999    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:36:57.372647 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.372629    2577 scope.go:117] "RemoveContainer" containerID="e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"
Sep 17 18:36:57.372836 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.372819    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-rbmlc"]
Sep 17 18:36:57.372956 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:57.372939    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c\": container with ID starting with e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c not found: ID does not exist" containerID="e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"
Sep 17 18:36:57.372996 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.372964    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c"} err="failed to get container status \"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c\": rpc error: code = NotFound desc = could not find container \"e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c\": container with ID starting with e6d468e3061a80edbe0d46a7c8a0eeea885492acf5421966d389833ddc679a7c not found: ID does not exist"
Sep 17 18:36:57.372996 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.372982    2577 scope.go:117] "RemoveContainer" containerID="889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"
Sep 17 18:36:57.373222 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:57.373206    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3\": container with ID starting with 889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3 not found: ID does not exist" containerID="889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"
Sep 17 18:36:57.373278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.373228    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3"} err="failed to get container status \"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3\": rpc error: code = NotFound desc = could not find container \"889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3\": container with ID starting with 889f68487355bf24d5be7d0018eca0f59600d0561239af7d0b3d5406baf0baa3 not found: ID does not exist"
Sep 17 18:36:57.373278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.373245    2577 scope.go:117] "RemoveContainer" containerID="aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"
Sep 17 18:36:57.373494 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:36:57.373477    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56\": container with ID starting with aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56 not found: ID does not exist" containerID="aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"
Sep 17 18:36:57.373545 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.373495    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56"} err="failed to get container status \"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56\": rpc error: code = NotFound desc = could not find container \"aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56\": container with ID starting with aad4bb7457816fad8eb484e90dd37fddf5f4ad949f6c2fceb4e99bc89c678c56 not found: ID does not exist"
Sep 17 18:36:57.389096 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.389074    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823643ac-34ed-4e6f-a35a-dcdac8286267-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:57.389170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.389098    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv67n\" (UniqueName: \"kubernetes.io/projected/823643ac-34ed-4e6f-a35a-dcdac8286267-kube-api-access-bv67n\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:57.389170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:57.389109    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/823643ac-34ed-4e6f-a35a-dcdac8286267-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:36:58.355018 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:58.354922    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerID="8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f" exitCode=0
Sep 17 18:36:58.355018 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:58.354992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerDied","Data":"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"}
Sep 17 18:36:58.941894 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:58.941860    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" path="/var/lib/kubelet/pods/823643ac-34ed-4e6f-a35a-dcdac8286267/volumes"
Sep 17 18:36:59.361766 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:59.361721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerStarted","Data":"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"}
Sep 17 18:36:59.362144 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:59.361803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerStarted","Data":"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"}
Sep 17 18:36:59.362144 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:59.361998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:36:59.382452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:36:59.382404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" podStartSLOduration=3.572886525 podStartE2EDuration="7.382390295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80950377s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:59.381155753 +0000 UTC m=+2430.705477177" watchObservedRunningTime="2026-09-17 18:36:59.382390295 +0000 UTC m=+2430.706711718"
Sep 17 18:37:00.366172 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:00.366145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:37:06.185737 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:06.185700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:06.376714 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:06.376685    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:37:06.653463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:06.653436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:36.380886 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:36.380852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:37:43.043352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.043316    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:37:43.043802 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.043569    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kserve-container" containerID="cri-o://cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3" gracePeriod=30
Sep 17 18:37:43.043802 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.043610    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kube-rbac-proxy" containerID="cri-o://fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343" gracePeriod=30
Sep 17 18:37:43.074968 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.074943    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:43.561427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.561398    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerID="fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343" exitCode=2
Sep 17 18:37:43.561604 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:43.561475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerDied","Data":"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"}
Sep 17 18:37:46.371872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:46.371819    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 17 18:37:46.378303 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:46.378273    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:37:48.192791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:48.192723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.623792 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:48.623768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:49.394016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.393993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:37:49.540212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location\") pod \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") "
Sep 17 18:37:49.540396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config\") pod \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") "
Sep 17 18:37:49.540396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") pod \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") "
Sep 17 18:37:49.540396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2rw\" (UniqueName: \"kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw\") pod \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") "
Sep 17 18:37:49.540396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert\") pod \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\" (UID: \"8bf46ca6-5485-44dc-931e-1f4f74cd6d70\") "
Sep 17 18:37:49.540611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bf46ca6-5485-44dc-931e-1f4f74cd6d70" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:49.540649 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bf46ca6-5485-44dc-931e-1f4f74cd6d70" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:49.540701 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.540682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bf46ca6-5485-44dc-931e-1f4f74cd6d70" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:49.542404 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.542382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bf46ca6-5485-44dc-931e-1f4f74cd6d70" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:49.542467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.542456    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw" (OuterVolumeSpecName: "kube-api-access-kp2rw") pod "8bf46ca6-5485-44dc-931e-1f4f74cd6d70" (UID: "8bf46ca6-5485-44dc-931e-1f4f74cd6d70"). InnerVolumeSpecName "kube-api-access-kp2rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:49.589483 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.589450    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerID="cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3" exitCode=0
Sep 17 18:37:49.589645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.589512    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"
Sep 17 18:37:49.589645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.589535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerDied","Data":"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"}
Sep 17 18:37:49.589645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.589607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7" event={"ID":"8bf46ca6-5485-44dc-931e-1f4f74cd6d70","Type":"ContainerDied","Data":"341a5d29909dec5b4c7e9e09272ab2c1075b05ee7a98bfa5f513e18add167b31"}
Sep 17 18:37:49.589645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.589641    2577 scope.go:117] "RemoveContainer" containerID="fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"
Sep 17 18:37:49.598583 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.598566    2577 scope.go:117] "RemoveContainer" containerID="cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"
Sep 17 18:37:49.605918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.605900    2577 scope.go:117] "RemoveContainer" containerID="8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"
Sep 17 18:37:49.612083 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.612062    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:37:49.613246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.613232    2577 scope.go:117] "RemoveContainer" containerID="fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"
Sep 17 18:37:49.613556 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:37:49.613514    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343\": container with ID starting with fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343 not found: ID does not exist" containerID="fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"
Sep 17 18:37:49.613674 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.613561    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343"} err="failed to get container status \"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343\": rpc error: code = NotFound desc = could not find container \"fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343\": container with ID starting with fc4b3ac4c01aec25bf02752771a618a1f987301529f018331f5476d48706a343 not found: ID does not exist"
Sep 17 18:37:49.613674 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.613585    2577 scope.go:117] "RemoveContainer" containerID="cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"
Sep 17 18:37:49.613969 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:37:49.613950    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3\": container with ID starting with cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3 not found: ID does not exist" containerID="cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"
Sep 17 18:37:49.614109 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.613978    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3"} err="failed to get container status \"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3\": rpc error: code = NotFound desc = could not find container \"cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3\": container with ID starting with cab379a1f0bc8e128735a7ef0dfedc5b9805f07f9c00a302127eb5ab2c6186f3 not found: ID does not exist"
Sep 17 18:37:49.614109 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.613999    2577 scope.go:117] "RemoveContainer" containerID="8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"
Sep 17 18:37:49.614311 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:37:49.614291    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f\": container with ID starting with 8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f not found: ID does not exist" containerID="8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"
Sep 17 18:37:49.614353 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.614319    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f"} err="failed to get container status \"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f\": rpc error: code = NotFound desc = could not find container \"8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f\": container with ID starting with 8e996d8726ef785af46ec2997d950634565f622cf000eb82834c4b085021240f not found: ID does not exist"
Sep 17 18:37:49.615592 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.615572    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hmcf7"]
Sep 17 18:37:49.641177 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.641157    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:37:49.641257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.641177    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:37:49.641257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.641188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2rw\" (UniqueName: \"kubernetes.io/projected/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kube-api-access-kp2rw\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:37:49.641257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.641197    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:37:49.641257 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:49.641206    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf46ca6-5485-44dc-931e-1f4f74cd6d70-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:37:50.941645 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:37:50.941611    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" path="/var/lib/kubelet/pods/8bf46ca6-5485-44dc-931e-1f4f74cd6d70/volumes"
Sep 17 18:38:26.813654 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:38:26.813606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:51.347026 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:38:51.346995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:56.402546 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:38:56.402506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:07.095512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:07.095479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:12.214957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.214926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:12.248777 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.248735    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:39:12.249072 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249074    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kserve-container"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kserve-container"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249101    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249107    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249114    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249119    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249131    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249144    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249156    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249164    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249170    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249179    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249184    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="storage-initializer"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249237    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249244    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="823643ac-34ed-4e6f-a35a-dcdac8286267" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249250    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249258    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kserve-container"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249264    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9496fd41-592c-4813-8df1-549ff1216fd8" containerName="kube-rbac-proxy"
Sep 17 18:39:12.249301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.249268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf46ca6-5485-44dc-931e-1f4f74cd6d70" containerName="kube-rbac-proxy"
Sep 17 18:39:12.252365 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.252350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.254844 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.254827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:12.254959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.254889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 18:39:12.254959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.254939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:39:12.255146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.255127    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.255285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.255149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:12.255285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.255177    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.255285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.255186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:39:12.261542 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.261519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:39:12.292022 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.291997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n4l\" (UniqueName: \"kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.292126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.292025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.292126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.292052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.292126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.292070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.292235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.292159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.392801 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.392772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.392958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.392837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n4l\" (UniqueName: \"kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.392958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.392863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.392958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.392905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.392958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.392930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.393289 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.393269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.393575 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.393552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.393671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.393551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.395366 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.395340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.400250 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.400230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n4l\" (UniqueName: \"kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.415180 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.415160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:12.562862 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.562768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:12.833615 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.833588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:39:12.942471 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.942439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerStarted","Data":"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"}
Sep 17 18:39:12.942628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:12.942491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerStarted","Data":"73230720d74ad63ffba616a7ae17862c3f283754faf9a87fcfcead712fd35d1d"}
Sep 17 18:39:16.954535 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:16.954497    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerID="1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7" exitCode=0
Sep 17 18:39:16.954958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:16.954563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerDied","Data":"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"}
Sep 17 18:39:16.955814 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:16.955794    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:39:18.869446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:18.869410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.903381 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.903350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.932743 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.932717    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:39:23.936199 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.936182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:23.938770 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.938732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:39:23.938872 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.938740    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 18:39:23.946847 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.946827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:39:23.994189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.994155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerStarted","Data":"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"}
Sep 17 18:39:23.994331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.994226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerStarted","Data":"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"}
Sep 17 18:39:23.994450 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.994432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:23.999868 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.999847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzlp\" (UniqueName: \"kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:23.999974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.999876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:23.999974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.999905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:23.999974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.999956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.000099 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:23.999985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.020088 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.020048    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podStartSLOduration=1.9991803 podStartE2EDuration="12.020036121s" totalImagesPullingTime="6.203890684s" totalInitContainerRuntime="3.816965137s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:24.018528591 +0000 UTC m=+2575.342850016" watchObservedRunningTime="2026-09-17 18:39:24.020036121 +0000 UTC m=+2575.344357544"
Sep 17 18:39:24.101162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzlp\" (UniqueName: \"kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.101268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.101337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.101453 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.101540 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:39:24.101489    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 17 18:39:24.101599 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:39:24.101564    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls podName:be3bf680-53cf-4160-84a5-e91bead01bd6 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:24.601543148 +0000 UTC m=+2575.925864563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 17 18:39:24.101599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.101951 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.102001 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.101955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.102037 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.102026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.112018 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.111999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzlp\" (UniqueName: \"kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.296116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.296040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:24.605956 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.605882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.608321 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.608298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.846115 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.846079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:24.998188 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.998160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:24.999442 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:24.999418    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:39:25.133213 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:25.133187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:39:25.134783 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:39:25.134747    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3bf680_53cf_4160_84a5_e91bead01bd6.slice/crio-559ad9717b78d174c0140eebcf0f29bbfd43b35c71f18871d598e12ee6b27091 WatchSource:0}: Error finding container 559ad9717b78d174c0140eebcf0f29bbfd43b35c71f18871d598e12ee6b27091: Status 404 returned error can't find the container with id 559ad9717b78d174c0140eebcf0f29bbfd43b35c71f18871d598e12ee6b27091
Sep 17 18:39:26.003998 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:26.003953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerStarted","Data":"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"}
Sep 17 18:39:26.004456 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:26.004026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerStarted","Data":"559ad9717b78d174c0140eebcf0f29bbfd43b35c71f18871d598e12ee6b27091"}
Sep 17 18:39:26.004456 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:26.004247    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:39:29.018218 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:29.018187    2577 generic.go:356] "Generic (PLEG): container finished" podID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerID="4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf" exitCode=0
Sep 17 18:39:29.018608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:29.018269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerDied","Data":"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"}
Sep 17 18:39:30.024741 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:30.024699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerStarted","Data":"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"}
Sep 17 18:39:30.025169 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:30.024776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerStarted","Data":"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"}
Sep 17 18:39:30.025169 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:30.025066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:30.025169 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:30.025117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:30.046297 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:30.046246    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podStartSLOduration=3.335680929 podStartE2EDuration="7.046230087s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.710549158s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:30.045273527 +0000 UTC m=+2581.369594950" watchObservedRunningTime="2026-09-17 18:39:30.046230087 +0000 UTC m=+2581.370551510"
Sep 17 18:39:31.008556 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:31.008525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:39:31.009137 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:31.009109    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:39:36.035254 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:36.035219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:39:41.009636 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:41.009597    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:39:51.009552 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:39:51.009514    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:01.009080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:01.009042    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:06.072516 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:06.072466    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:40:11.009569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:11.009530    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:16.038298 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:16.038267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:40:21.009855 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:21.009816    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:23.994213 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:23.994183    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:40:23.994629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:23.994403    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container" containerID="cri-o://76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3" gracePeriod=30
Sep 17 18:40:23.994629 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:23.994454    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy" containerID="cri-o://14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077" gracePeriod=30
Sep 17 18:40:24.023008 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:24.022982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:24.271874 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:24.271788    2577 generic.go:356] "Generic (PLEG): container finished" podID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerID="14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077" exitCode=2
Sep 17 18:40:24.271874 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:24.271841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerDied","Data":"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"}
Sep 17 18:40:26.030214 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:26.030171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 18:40:27.078029 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:27.077965    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:40:29.150197 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:29.150169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:29.492052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:29.491964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:31.009630 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.009594    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:31.030961 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.030913    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 18:40:31.137334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.137311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:40:31.271231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert\") pod \"be3bf680-53cf-4160-84a5-e91bead01bd6\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") "
Sep 17 18:40:31.271231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtzlp\" (UniqueName: \"kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp\") pod \"be3bf680-53cf-4160-84a5-e91bead01bd6\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") "
Sep 17 18:40:31.271441 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") pod \"be3bf680-53cf-4160-84a5-e91bead01bd6\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") "
Sep 17 18:40:31.271441 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location\") pod \"be3bf680-53cf-4160-84a5-e91bead01bd6\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") "
Sep 17 18:40:31.271441 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config\") pod \"be3bf680-53cf-4160-84a5-e91bead01bd6\" (UID: \"be3bf680-53cf-4160-84a5-e91bead01bd6\") "
Sep 17 18:40:31.271646 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be3bf680-53cf-4160-84a5-e91bead01bd6" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:31.271700 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be3bf680-53cf-4160-84a5-e91bead01bd6" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:31.271814 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.271779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be3bf680-53cf-4160-84a5-e91bead01bd6" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:31.273275 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.273256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be3bf680-53cf-4160-84a5-e91bead01bd6" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:31.273446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.273426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp" (OuterVolumeSpecName: "kube-api-access-xtzlp") pod "be3bf680-53cf-4160-84a5-e91bead01bd6" (UID: "be3bf680-53cf-4160-84a5-e91bead01bd6"). InnerVolumeSpecName "kube-api-access-xtzlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:31.303454 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.303427    2577 generic.go:356] "Generic (PLEG): container finished" podID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerID="76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3" exitCode=0
Sep 17 18:40:31.303549 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.303477    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"
Sep 17 18:40:31.303549 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.303512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerDied","Data":"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"}
Sep 17 18:40:31.303633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.303566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr" event={"ID":"be3bf680-53cf-4160-84a5-e91bead01bd6","Type":"ContainerDied","Data":"559ad9717b78d174c0140eebcf0f29bbfd43b35c71f18871d598e12ee6b27091"}
Sep 17 18:40:31.303633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.303592    2577 scope.go:117] "RemoveContainer" containerID="14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"
Sep 17 18:40:31.311942 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.311925    2577 scope.go:117] "RemoveContainer" containerID="76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"
Sep 17 18:40:31.321155 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.321132    2577 scope.go:117] "RemoveContainer" containerID="4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"
Sep 17 18:40:31.326400 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.326379    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:40:31.329726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.329708    2577 scope.go:117] "RemoveContainer" containerID="14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"
Sep 17 18:40:31.330009 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.329992    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jjxzr"]
Sep 17 18:40:31.330118 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:40:31.330103    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077\": container with ID starting with 14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077 not found: ID does not exist" containerID="14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"
Sep 17 18:40:31.330166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.330126    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077"} err="failed to get container status \"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077\": rpc error: code = NotFound desc = could not find container \"14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077\": container with ID starting with 14dac085de6254cad436599618f58d662dc729021731b89965f18117f44c4077 not found: ID does not exist"
Sep 17 18:40:31.330166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.330149    2577 scope.go:117] "RemoveContainer" containerID="76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"
Sep 17 18:40:31.330424 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:40:31.330406    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3\": container with ID starting with 76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3 not found: ID does not exist" containerID="76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"
Sep 17 18:40:31.330489 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.330433    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3"} err="failed to get container status \"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3\": rpc error: code = NotFound desc = could not find container \"76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3\": container with ID starting with 76218c834a9b167b4f3c99a26373f97dc8767b03c5fed784055d9618891b3dd3 not found: ID does not exist"
Sep 17 18:40:31.330489 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.330457    2577 scope.go:117] "RemoveContainer" containerID="4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"
Sep 17 18:40:31.330738 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:40:31.330722    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf\": container with ID starting with 4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf not found: ID does not exist" containerID="4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"
Sep 17 18:40:31.330903 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.330744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf"} err="failed to get container status \"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf\": rpc error: code = NotFound desc = could not find container \"4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf\": container with ID starting with 4677288e874b58f48a72c9914b122c3afec9745d2703dc4eef02c4676b03d6bf not found: ID does not exist"
Sep 17 18:40:31.372215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.372189    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:40:31.372215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.372217    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf680-53cf-4160-84a5-e91bead01bd6-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:40:31.372357 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.372227    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzlp\" (UniqueName: \"kubernetes.io/projected/be3bf680-53cf-4160-84a5-e91bead01bd6-kube-api-access-xtzlp\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:40:31.372357 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.372236    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be3bf680-53cf-4160-84a5-e91bead01bd6-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:40:31.372357 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:31.372244    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be3bf680-53cf-4160-84a5-e91bead01bd6-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:40:32.941527 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:32.941489    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" path="/var/lib/kubelet/pods/be3bf680-53cf-4160-84a5-e91bead01bd6/volumes"
Sep 17 18:40:41.009509 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:41.009466    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:51.009637 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:51.009600    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:40:54.941908 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:40:54.941874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:41:03.673170 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:03.673131    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:41:03.673679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:03.673417    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" containerID="cri-o://6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62" gracePeriod=30
Sep 17 18:41:03.673679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:03.673417    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kube-rbac-proxy" containerID="cri-o://cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a" gracePeriod=30
Sep 17 18:41:03.700050 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:03.700025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:04.446287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:04.446256    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerID="cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a" exitCode=2
Sep 17 18:41:04.446615 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:04.446329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerDied","Data":"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"}
Sep 17 18:41:04.936703 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:04.936658    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:41:06.005283 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:06.005245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 18:41:08.861811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.861772    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:08.885012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.884984    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:41:08.885349 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy"
Sep 17 18:41:08.885396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885350    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy"
Sep 17 18:41:08.885396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885361    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container"
Sep 17 18:41:08.885396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container"
Sep 17 18:41:08.885396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="storage-initializer"
Sep 17 18:41:08.885396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="storage-initializer"
Sep 17 18:41:08.885548 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885434    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kserve-container"
Sep 17 18:41:08.885548 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.885443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3bf680-53cf-4160-84a5-e91bead01bd6" containerName="kube-rbac-proxy"
Sep 17 18:41:08.888421 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.888407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:08.890852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.890832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:41:08.890955 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.890850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 18:41:08.898895 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.898875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:41:08.982426 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.982394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smgn\" (UniqueName: \"kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:08.982601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.982472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:08.982601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.982501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:08.982601 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.982566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:08.982717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:08.982597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.066966 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.066933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:09.084031 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084147 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084147 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084147 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084293 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smgn\" (UniqueName: \"kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084633 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084729 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.084852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.084832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.086598 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.086569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.092406 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.092367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smgn\" (UniqueName: \"kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.110797 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.110779    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:41:09.184957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.184875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location\") pod \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") "
Sep 17 18:41:09.184957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.184929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8n4l\" (UniqueName: \"kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l\") pod \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") "
Sep 17 18:41:09.184957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.184956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config\") pod \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") "
Sep 17 18:41:09.185244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.184981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert\") pod \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") "
Sep 17 18:41:09.185244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.185045    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls\") pod \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\" (UID: \"f4120c13-2fb1-425d-a852-a01fe6ffdd9a\") "
Sep 17 18:41:09.185359 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.185252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4120c13-2fb1-425d-a852-a01fe6ffdd9a" (UID: "f4120c13-2fb1-425d-a852-a01fe6ffdd9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:09.185416 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.185373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4120c13-2fb1-425d-a852-a01fe6ffdd9a" (UID: "f4120c13-2fb1-425d-a852-a01fe6ffdd9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:09.185416 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.185386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4120c13-2fb1-425d-a852-a01fe6ffdd9a" (UID: "f4120c13-2fb1-425d-a852-a01fe6ffdd9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:09.187056 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.187033    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l" (OuterVolumeSpecName: "kube-api-access-g8n4l") pod "f4120c13-2fb1-425d-a852-a01fe6ffdd9a" (UID: "f4120c13-2fb1-425d-a852-a01fe6ffdd9a"). InnerVolumeSpecName "kube-api-access-g8n4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:09.187337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.187307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4120c13-2fb1-425d-a852-a01fe6ffdd9a" (UID: "f4120c13-2fb1-425d-a852-a01fe6ffdd9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:09.198528 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.198498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:09.286726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.286694    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.286726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.286722    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.286924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.286735    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n4l\" (UniqueName: \"kubernetes.io/projected/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-api-access-g8n4l\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.286924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.286748    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.286924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.286775    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4120c13-2fb1-425d-a852-a01fe6ffdd9a-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.468475 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.468447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:41:09.470224 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:41:09.470186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee02d60_ae68_4ce4_8bb0_51e9e460cade.slice/crio-b81b67daf8f5058ed579c8da137a97fb8f2d7301edaa9be6e95eafd98d695f45 WatchSource:0}: Error finding container b81b67daf8f5058ed579c8da137a97fb8f2d7301edaa9be6e95eafd98d695f45: Status 404 returned error can't find the container with id b81b67daf8f5058ed579c8da137a97fb8f2d7301edaa9be6e95eafd98d695f45
Sep 17 18:41:09.470492 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.470463    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerID="6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62" exitCode=0
Sep 17 18:41:09.470573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.470504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerDied","Data":"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"}
Sep 17 18:41:09.470573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.470546    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"
Sep 17 18:41:09.470573 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.470564    2577 scope.go:117] "RemoveContainer" containerID="cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"
Sep 17 18:41:09.470714 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.470552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l" event={"ID":"f4120c13-2fb1-425d-a852-a01fe6ffdd9a","Type":"ContainerDied","Data":"73230720d74ad63ffba616a7ae17862c3f283754faf9a87fcfcead712fd35d1d"}
Sep 17 18:41:09.479474 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.479451    2577 scope.go:117] "RemoveContainer" containerID="6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"
Sep 17 18:41:09.499372 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.499352    2577 scope.go:117] "RemoveContainer" containerID="1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"
Sep 17 18:41:09.511016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.510993    2577 scope.go:117] "RemoveContainer" containerID="cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"
Sep 17 18:41:09.511619 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:41:09.511419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a\": container with ID starting with cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a not found: ID does not exist" containerID="cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"
Sep 17 18:41:09.511619 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.511462    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a"} err="failed to get container status \"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a\": rpc error: code = NotFound desc = could not find container \"cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a\": container with ID starting with cf6808d87c958b4b7a7e6fd2eb26c38c3917e7698ba956d44fdfa5827db7c28a not found: ID does not exist"
Sep 17 18:41:09.511619 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.511488    2577 scope.go:117] "RemoveContainer" containerID="6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"
Sep 17 18:41:09.512004 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:41:09.511986    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62\": container with ID starting with 6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62 not found: ID does not exist" containerID="6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"
Sep 17 18:41:09.512069 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.512012    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62"} err="failed to get container status \"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62\": rpc error: code = NotFound desc = could not find container \"6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62\": container with ID starting with 6c39681a5d1e659042b4bf303db6ca923c8578d976a26b751a7b8099555e3c62 not found: ID does not exist"
Sep 17 18:41:09.512069 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.512028    2577 scope.go:117] "RemoveContainer" containerID="1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"
Sep 17 18:41:09.512305 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:41:09.512279    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7\": container with ID starting with 1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7 not found: ID does not exist" containerID="1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"
Sep 17 18:41:09.512413 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.512310    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7"} err="failed to get container status \"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7\": rpc error: code = NotFound desc = could not find container \"1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7\": container with ID starting with 1272731bf60376decf57641a08f4d558340bedbbaa6f01efb365717ae949e4d7 not found: ID does not exist"
Sep 17 18:41:09.519586 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.519564    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:41:09.525115 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:09.525092    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-mds8l"]
Sep 17 18:41:10.476978 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:10.476937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerStarted","Data":"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"}
Sep 17 18:41:10.477427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:10.476999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerStarted","Data":"b81b67daf8f5058ed579c8da137a97fb8f2d7301edaa9be6e95eafd98d695f45"}
Sep 17 18:41:10.941873 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:10.941839    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" path="/var/lib/kubelet/pods/f4120c13-2fb1-425d-a852-a01fe6ffdd9a/volumes"
Sep 17 18:41:14.502228 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:14.502196    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerID="2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f" exitCode=0
Sep 17 18:41:14.502641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:14.502277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerDied","Data":"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"}
Sep 17 18:41:15.509319 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:15.509273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerStarted","Data":"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"}
Sep 17 18:41:15.509718 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:15.509348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerStarted","Data":"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"}
Sep 17 18:41:15.509718 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:15.509668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:15.509818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:15.509720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:15.530457 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:15.530401    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" podStartSLOduration=3.540651347 podStartE2EDuration="7.530380403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.989729056s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:15.53010224 +0000 UTC m=+2686.854423667" watchObservedRunningTime="2026-09-17 18:41:15.530380403 +0000 UTC m=+2686.854701826"
Sep 17 18:41:21.520197 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:21.520171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:49.960614 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:49.960583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:51.524286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:51.524256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:41:55.135261 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:55.135229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:55.323364 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:55.323333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:58.932846 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:58.932812    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:41:58.933288 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:58.933050    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kserve-container" containerID="cri-o://2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4" gracePeriod=30
Sep 17 18:41:58.933288 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:58.933099    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kube-rbac-proxy" containerID="cri-o://cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8" gracePeriod=30
Sep 17 18:41:58.964248 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:58.964217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:59.710721 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:59.710687    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerID="cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8" exitCode=2
Sep 17 18:41:59.710953 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:59.710777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerDied","Data":"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"}
Sep 17 18:41:59.978939 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:41:59.978910    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:42:00.016466 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert\") pod \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") "
Sep 17 18:42:00.016642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7smgn\" (UniqueName: \"kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn\") pod \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") "
Sep 17 18:42:00.016642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls\") pod \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") "
Sep 17 18:42:00.016642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location\") pod \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") "
Sep 17 18:42:00.016642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config\") pod \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\" (UID: \"4ee02d60-ae68-4ce4-8bb0-51e9e460cade\") "
Sep 17 18:42:00.016923 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ee02d60-ae68-4ce4-8bb0-51e9e460cade" (UID: "4ee02d60-ae68-4ce4-8bb0-51e9e460cade"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:00.016979 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ee02d60-ae68-4ce4-8bb0-51e9e460cade" (UID: "4ee02d60-ae68-4ce4-8bb0-51e9e460cade"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:00.017018 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.016997    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ee02d60-ae68-4ce4-8bb0-51e9e460cade" (UID: "4ee02d60-ae68-4ce4-8bb0-51e9e460cade"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:00.018684 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.018662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ee02d60-ae68-4ce4-8bb0-51e9e460cade" (UID: "4ee02d60-ae68-4ce4-8bb0-51e9e460cade"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:00.018824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.018801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn" (OuterVolumeSpecName: "kube-api-access-7smgn") pod "4ee02d60-ae68-4ce4-8bb0-51e9e460cade" (UID: "4ee02d60-ae68-4ce4-8bb0-51e9e460cade"). InnerVolumeSpecName "kube-api-access-7smgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:00.117379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.117327    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.117379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.117369    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7smgn\" (UniqueName: \"kubernetes.io/projected/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-api-access-7smgn\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.117379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.117380    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.117379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.117390    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.117379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.117399    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee02d60-ae68-4ce4-8bb0-51e9e460cade-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.717519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.717487    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerID="2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4" exitCode=0
Sep 17 18:42:00.717726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.717541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"
Sep 17 18:42:00.717726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.717572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerDied","Data":"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"}
Sep 17 18:42:00.717726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.717642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g" event={"ID":"4ee02d60-ae68-4ce4-8bb0-51e9e460cade","Type":"ContainerDied","Data":"b81b67daf8f5058ed579c8da137a97fb8f2d7301edaa9be6e95eafd98d695f45"}
Sep 17 18:42:00.717726 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.717679    2577 scope.go:117] "RemoveContainer" containerID="cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"
Sep 17 18:42:00.727236 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.727217    2577 scope.go:117] "RemoveContainer" containerID="2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"
Sep 17 18:42:00.735162 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.735135    2577 scope.go:117] "RemoveContainer" containerID="2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"
Sep 17 18:42:00.740180 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.740161    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:42:00.743080 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.743064    2577 scope.go:117] "RemoveContainer" containerID="cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"
Sep 17 18:42:00.743349 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:00.743330    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8\": container with ID starting with cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8 not found: ID does not exist" containerID="cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"
Sep 17 18:42:00.743418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.743361    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8"} err="failed to get container status \"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8\": rpc error: code = NotFound desc = could not find container \"cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8\": container with ID starting with cbdaae538d06a8801840c787fc7a22f0c016ea45b28df961863e18671734f4c8 not found: ID does not exist"
Sep 17 18:42:00.743418 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.743390    2577 scope.go:117] "RemoveContainer" containerID="2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"
Sep 17 18:42:00.743657 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:00.743624    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4\": container with ID starting with 2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4 not found: ID does not exist" containerID="2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"
Sep 17 18:42:00.743657 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.743648    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4"} err="failed to get container status \"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4\": rpc error: code = NotFound desc = could not find container \"2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4\": container with ID starting with 2b59adb8336edd072eaadbf29bed9ffca4f466911d5d29ed40c430006592c4c4 not found: ID does not exist"
Sep 17 18:42:00.743870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.743666    2577 scope.go:117] "RemoveContainer" containerID="2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"
Sep 17 18:42:00.744296 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:00.744023    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f\": container with ID starting with 2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f not found: ID does not exist" containerID="2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"
Sep 17 18:42:00.744296 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.744060    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f"} err="failed to get container status \"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f\": rpc error: code = NotFound desc = could not find container \"2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f\": container with ID starting with 2c6486306b0c8aecfd35f224fb1c25b0a48cd39db0537c5dc08b24ecc0aa8c5f not found: ID does not exist"
Sep 17 18:42:00.745788 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.745769    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jnm9g"]
Sep 17 18:42:00.942054 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:00.942024    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" path="/var/lib/kubelet/pods/4ee02d60-ae68-4ce4-8bb0-51e9e460cade/volumes"
Sep 17 18:42:04.082127 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.082094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:04.109972 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.109942    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:04.110278 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110267    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="storage-initializer"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110280    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="storage-initializer"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="storage-initializer"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="storage-initializer"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110302    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kube-rbac-proxy"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110309    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kube-rbac-proxy"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110317    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kube-rbac-proxy"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110322    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kube-rbac-proxy"
Sep 17 18:42:04.110329 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110331    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110336    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110410    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110418    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4120c13-2fb1-425d-a852-a01fe6ffdd9a" containerName="kube-rbac-proxy"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110424    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kserve-container"
Sep 17 18:42:04.110628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.110433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee02d60-ae68-4ce4-8bb0-51e9e460cade" containerName="kube-rbac-proxy"
Sep 17 18:42:04.113389 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.113373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.116151 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116099    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:42:04.116374 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 18:42:04.116463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:42:04.116463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116384    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:42:04.116463 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:42:04.116973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.116957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:42:04.117086 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.117060    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:42:04.124564 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.124542    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:04.247533 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.247493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:04.253617 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.253595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.253722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.253627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.253722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.253646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qgs\" (UniqueName: \"kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.253865 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.253799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.253918 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.253867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.354917 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.354824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.354917 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.354875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.354917 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.354909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.355196 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.354927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.355196 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.354943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qgs\" (UniqueName: \"kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.355356 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.355336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.355680 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.355656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.355818 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.355656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.357377 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.357358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.363636 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.363615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qgs\" (UniqueName: \"kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.423793 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.423734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:04.696974 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.696953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:04.699413 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:42:04.699384    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96693906_81e7_489e_a534_d42d117f2ac3.slice/crio-4135aff8d109a04e980ccf39fd4a7f446afc2aebfe4ee1f2ff5440e25f56dea5 WatchSource:0}: Error finding container 4135aff8d109a04e980ccf39fd4a7f446afc2aebfe4ee1f2ff5440e25f56dea5: Status 404 returned error can't find the container with id 4135aff8d109a04e980ccf39fd4a7f446afc2aebfe4ee1f2ff5440e25f56dea5
Sep 17 18:42:04.737566 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:04.737539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerStarted","Data":"4135aff8d109a04e980ccf39fd4a7f446afc2aebfe4ee1f2ff5440e25f56dea5"}
Sep 17 18:42:05.743339 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:05.743301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerStarted","Data":"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"}
Sep 17 18:42:08.756958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:08.756922    2577 generic.go:356] "Generic (PLEG): container finished" podID="96693906-81e7-489e-a534-d42d117f2ac3" containerID="0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78" exitCode=0
Sep 17 18:42:08.757299 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:08.756967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerDied","Data":"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"}
Sep 17 18:42:09.763145 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.763109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerStarted","Data":"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"}
Sep 17 18:42:09.763582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.763171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerStarted","Data":"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"}
Sep 17 18:42:09.763582 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.763443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:09.763691 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.763609    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:09.764769 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.764733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:42:09.784338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:09.784288    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podStartSLOduration=1.944062116 podStartE2EDuration="5.784274388s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.840212272s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:09.782395567 +0000 UTC m=+2741.106717012" watchObservedRunningTime="2026-09-17 18:42:09.784274388 +0000 UTC m=+2741.108595810"
Sep 17 18:42:10.768123 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:10.768073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:42:15.773116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:15.773087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:15.773688 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:15.773664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:42:25.774375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:25.774346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:34.206854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.206818    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:34.207368 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.207038    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container" containerID="cri-o://6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b" gracePeriod=30
Sep 17 18:42:34.207368 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.207122    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kube-rbac-proxy" containerID="cri-o://7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0" gracePeriod=30
Sep 17 18:42:34.230157 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.230131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:34.852631 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.852608    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:34.874677 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874649    2577 generic.go:356] "Generic (PLEG): container finished" podID="96693906-81e7-489e-a534-d42d117f2ac3" containerID="7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0" exitCode=2
Sep 17 18:42:34.874677 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874673    2577 generic.go:356] "Generic (PLEG): container finished" podID="96693906-81e7-489e-a534-d42d117f2ac3" containerID="6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b" exitCode=0
Sep 17 18:42:34.874932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"
Sep 17 18:42:34.874932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerDied","Data":"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"}
Sep 17 18:42:34.874932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerDied","Data":"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"}
Sep 17 18:42:34.874932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn" event={"ID":"96693906-81e7-489e-a534-d42d117f2ac3","Type":"ContainerDied","Data":"4135aff8d109a04e980ccf39fd4a7f446afc2aebfe4ee1f2ff5440e25f56dea5"}
Sep 17 18:42:34.874932 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.874879    2577 scope.go:117] "RemoveContainer" containerID="7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"
Sep 17 18:42:34.886461 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.886441    2577 scope.go:117] "RemoveContainer" containerID="6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"
Sep 17 18:42:34.894093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.894068    2577 scope.go:117] "RemoveContainer" containerID="0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"
Sep 17 18:42:34.901605 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.901589    2577 scope.go:117] "RemoveContainer" containerID="7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"
Sep 17 18:42:34.901895 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:34.901878    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0\": container with ID starting with 7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0 not found: ID does not exist" containerID="7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"
Sep 17 18:42:34.901957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.901902    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"} err="failed to get container status \"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0\": rpc error: code = NotFound desc = could not find container \"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0\": container with ID starting with 7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0 not found: ID does not exist"
Sep 17 18:42:34.901957 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.901918    2577 scope.go:117] "RemoveContainer" containerID="6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"
Sep 17 18:42:34.902153 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:34.902136    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b\": container with ID starting with 6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b not found: ID does not exist" containerID="6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"
Sep 17 18:42:34.902190 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"} err="failed to get container status \"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b\": rpc error: code = NotFound desc = could not find container \"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b\": container with ID starting with 6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b not found: ID does not exist"
Sep 17 18:42:34.902190 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902180    2577 scope.go:117] "RemoveContainer" containerID="0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"
Sep 17 18:42:34.902396 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:34.902381    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78\": container with ID starting with 0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78 not found: ID does not exist" containerID="0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"
Sep 17 18:42:34.902435 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902400    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"} err="failed to get container status \"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78\": rpc error: code = NotFound desc = could not find container \"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78\": container with ID starting with 0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78 not found: ID does not exist"
Sep 17 18:42:34.902435 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902413    2577 scope.go:117] "RemoveContainer" containerID="7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"
Sep 17 18:42:34.902671 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902654    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0"} err="failed to get container status \"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0\": rpc error: code = NotFound desc = could not find container \"7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0\": container with ID starting with 7c96b244d3b0c9ad2f6584db47e258226fec5e2932ccdaf976c47ffc957049a0 not found: ID does not exist"
Sep 17 18:42:34.902716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902672    2577 scope.go:117] "RemoveContainer" containerID="6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"
Sep 17 18:42:34.902959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902942    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b"} err="failed to get container status \"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b\": rpc error: code = NotFound desc = could not find container \"6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b\": container with ID starting with 6fb52b05bebcb4058e7971ac1593e7b2b553ce2049184e234e6f53e1c4569f7b not found: ID does not exist"
Sep 17 18:42:34.903014 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.902960    2577 scope.go:117] "RemoveContainer" containerID="0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"
Sep 17 18:42:34.903183 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.903162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78"} err="failed to get container status \"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78\": rpc error: code = NotFound desc = could not find container \"0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78\": container with ID starting with 0a22cba9b97d7d5908b256c2c9211fc18d59dc644fd77f2c304d968a8756fa78 not found: ID does not exist"
Sep 17 18:42:34.915576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location\") pod \"96693906-81e7-489e-a534-d42d117f2ac3\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") "
Sep 17 18:42:34.915670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config\") pod \"96693906-81e7-489e-a534-d42d117f2ac3\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") "
Sep 17 18:42:34.915670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls\") pod \"96693906-81e7-489e-a534-d42d117f2ac3\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") "
Sep 17 18:42:34.915799 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5qgs\" (UniqueName: \"kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs\") pod \"96693906-81e7-489e-a534-d42d117f2ac3\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") "
Sep 17 18:42:34.915799 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert\") pod \"96693906-81e7-489e-a534-d42d117f2ac3\" (UID: \"96693906-81e7-489e-a534-d42d117f2ac3\") "
Sep 17 18:42:34.915919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96693906-81e7-489e-a534-d42d117f2ac3" (UID: "96693906-81e7-489e-a534-d42d117f2ac3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:34.916014 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.915985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96693906-81e7-489e-a534-d42d117f2ac3" (UID: "96693906-81e7-489e-a534-d42d117f2ac3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:34.916014 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.916014    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96693906-81e7-489e-a534-d42d117f2ac3-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:34.916217 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.916157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96693906-81e7-489e-a534-d42d117f2ac3" (UID: "96693906-81e7-489e-a534-d42d117f2ac3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:34.917670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.917646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96693906-81e7-489e-a534-d42d117f2ac3" (UID: "96693906-81e7-489e-a534-d42d117f2ac3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:34.917778 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:34.917730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs" (OuterVolumeSpecName: "kube-api-access-m5qgs") pod "96693906-81e7-489e-a534-d42d117f2ac3" (UID: "96693906-81e7-489e-a534-d42d117f2ac3"). InnerVolumeSpecName "kube-api-access-m5qgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:35.016717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.016686    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qgs\" (UniqueName: \"kubernetes.io/projected/96693906-81e7-489e-a534-d42d117f2ac3-kube-api-access-m5qgs\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:35.016717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.016714    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:35.016717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.016723    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96693906-81e7-489e-a534-d42d117f2ac3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:35.016952 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.016731    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96693906-81e7-489e-a534-d42d117f2ac3-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:42:35.191350 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.191315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:35.194718 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:35.194693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-2fxmn"]
Sep 17 18:42:36.942470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:36.942433    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96693906-81e7-489e-a534-d42d117f2ac3" path="/var/lib/kubelet/pods/96693906-81e7-489e-a534-d42d117f2ac3/volumes"
Sep 17 18:42:39.369553 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.369521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:39.402173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:42:39.402476 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402464    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="storage-initializer"
Sep 17 18:42:39.402522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="storage-initializer"
Sep 17 18:42:39.402522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402495    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container"
Sep 17 18:42:39.402522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container"
Sep 17 18:42:39.402522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402510    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kube-rbac-proxy"
Sep 17 18:42:39.402522 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kube-rbac-proxy"
Sep 17 18:42:39.402686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402570    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kserve-container"
Sep 17 18:42:39.402686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.402589    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96693906-81e7-489e-a534-d42d117f2ac3" containerName="kube-rbac-proxy"
Sep 17 18:42:39.407076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.407059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.409494 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.409455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 18:42:39.409679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.409576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:42:39.409679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.409626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:42:39.409898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.409678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:42:39.409898 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.409879    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:42:39.410420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.410403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:42:39.410516 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.410434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:42:39.415385 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.415364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:42:39.455518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.455492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.455628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.455520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.455628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.455542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlft\" (UniqueName: \"kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.455707 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.455648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.455707 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.455678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.556652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556686 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.556690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556903 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.556739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556903 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.556784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556903 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.556817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlft\" (UniqueName: \"kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.556903 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:39.556882    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:42:39.557060 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:42:39.556954    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls podName:edb3af0c-d9c7-4847-87cc-477cea8d791b nodeName:}" failed. No retries permitted until 2026-09-17 18:42:40.056935497 +0000 UTC m=+2771.381256898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:42:39.557156 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.557136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.557576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.557555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.557612 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.557555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.565800 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.565777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlft\" (UniqueName: \"kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:39.746962 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:39.746884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:40.060610 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.060527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:40.063017 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.062993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:40.318325 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.318216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:40.589670 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.589531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:42:40.592016 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:42:40.591992    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb3af0c_d9c7_4847_87cc_477cea8d791b.slice/crio-3c6d5361cb75ce590d8c2786405ac61716ba794dcf3ab0353570a96bfb645530 WatchSource:0}: Error finding container 3c6d5361cb75ce590d8c2786405ac61716ba794dcf3ab0353570a96bfb645530: Status 404 returned error can't find the container with id 3c6d5361cb75ce590d8c2786405ac61716ba794dcf3ab0353570a96bfb645530
Sep 17 18:42:40.909822 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.909782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerStarted","Data":"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"}
Sep 17 18:42:40.910027 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:40.909852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerStarted","Data":"3c6d5361cb75ce590d8c2786405ac61716ba794dcf3ab0353570a96bfb645530"}
Sep 17 18:42:45.932581 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:45.932548    2577 generic.go:356] "Generic (PLEG): container finished" podID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerID="462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47" exitCode=0
Sep 17 18:42:45.932982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:45.932603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerDied","Data":"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"}
Sep 17 18:42:46.947725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:46.947689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerStarted","Data":"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"}
Sep 17 18:42:46.948203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:46.947745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerStarted","Data":"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"}
Sep 17 18:42:46.948203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:46.948029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:46.948203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:46.948104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:42:46.967998 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:46.967924    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" podStartSLOduration=3.63249356 podStartE2EDuration="7.967876941s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.335383381s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:46.967081168 +0000 UTC m=+2778.291402614" watchObservedRunningTime="2026-09-17 18:42:46.967876941 +0000 UTC m=+2778.292198367"
Sep 17 18:42:52.956802 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:42:52.956773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:43:22.961251 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:22.961220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:43:25.271279 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:25.271247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:25.408844 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:25.408813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:25.605964 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:25.605881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:29.471447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:29.471416    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:43:29.471892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:29.471641    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kserve-container" containerID="cri-o://4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14" gracePeriod=30
Sep 17 18:43:29.471892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:29.471673    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kube-rbac-proxy" containerID="cri-o://39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e" gracePeriod=30
Sep 17 18:43:29.494960 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:29.494935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:30.133104 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.133068    2577 generic.go:356] "Generic (PLEG): container finished" podID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerID="39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e" exitCode=2
Sep 17 18:43:30.133285 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.133143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerDied","Data":"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"}
Sep 17 18:43:30.462856 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:43:30.462821    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb3af0c_d9c7_4847_87cc_477cea8d791b.slice/crio-4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb3af0c_d9c7_4847_87cc_477cea8d791b.slice/crio-conmon-4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:43:30.462983 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:43:30.462925    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb3af0c_d9c7_4847_87cc_477cea8d791b.slice/crio-4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:43:30.623126 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.623101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:43:30.678971 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.678945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") pod \"edb3af0c-d9c7-4847-87cc-477cea8d791b\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") "
Sep 17 18:43:30.679146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.678981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert\") pod \"edb3af0c-d9c7-4847-87cc-477cea8d791b\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") "
Sep 17 18:43:30.679146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location\") pod \"edb3af0c-d9c7-4847-87cc-477cea8d791b\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") "
Sep 17 18:43:30.679146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjlft\" (UniqueName: \"kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft\") pod \"edb3af0c-d9c7-4847-87cc-477cea8d791b\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") "
Sep 17 18:43:30.679146 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config\") pod \"edb3af0c-d9c7-4847-87cc-477cea8d791b\" (UID: \"edb3af0c-d9c7-4847-87cc-477cea8d791b\") "
Sep 17 18:43:30.679378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edb3af0c-d9c7-4847-87cc-477cea8d791b" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:30.679452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edb3af0c-d9c7-4847-87cc-477cea8d791b" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:30.679519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.679503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edb3af0c-d9c7-4847-87cc-477cea8d791b" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:30.681111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.681091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edb3af0c-d9c7-4847-87cc-477cea8d791b" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:30.681185 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.681132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft" (OuterVolumeSpecName: "kube-api-access-wjlft") pod "edb3af0c-d9c7-4847-87cc-477cea8d791b" (UID: "edb3af0c-d9c7-4847-87cc-477cea8d791b"). InnerVolumeSpecName "kube-api-access-wjlft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:30.780262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.780182    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlft\" (UniqueName: \"kubernetes.io/projected/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-api-access-wjlft\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.780262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.780208    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.780262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.780217    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb3af0c-d9c7-4847-87cc-477cea8d791b-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.780262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.780226    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb3af0c-d9c7-4847-87cc-477cea8d791b-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.780262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:30.780234    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb3af0c-d9c7-4847-87cc-477cea8d791b-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:43:31.139725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.139698    2577 generic.go:356] "Generic (PLEG): container finished" podID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerID="4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14" exitCode=0
Sep 17 18:43:31.139889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.139774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"
Sep 17 18:43:31.139889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.139793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerDied","Data":"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"}
Sep 17 18:43:31.139889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.139856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z" event={"ID":"edb3af0c-d9c7-4847-87cc-477cea8d791b","Type":"ContainerDied","Data":"3c6d5361cb75ce590d8c2786405ac61716ba794dcf3ab0353570a96bfb645530"}
Sep 17 18:43:31.139889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.139884    2577 scope.go:117] "RemoveContainer" containerID="39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"
Sep 17 18:43:31.147929 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.147906    2577 scope.go:117] "RemoveContainer" containerID="4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"
Sep 17 18:43:31.155490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.155467    2577 scope.go:117] "RemoveContainer" containerID="462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"
Sep 17 18:43:31.158177 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.158128    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:43:31.164208 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.163038    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-5wr2z"]
Sep 17 18:43:31.169396 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.169375    2577 scope.go:117] "RemoveContainer" containerID="39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"
Sep 17 18:43:31.169690 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:43:31.169666    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e\": container with ID starting with 39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e not found: ID does not exist" containerID="39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"
Sep 17 18:43:31.169791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.169701    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e"} err="failed to get container status \"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e\": rpc error: code = NotFound desc = could not find container \"39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e\": container with ID starting with 39eb34221a853b47710e21ec13a5f877434f7a907e0dfd17e890cd70c57f324e not found: ID does not exist"
Sep 17 18:43:31.169791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.169725    2577 scope.go:117] "RemoveContainer" containerID="4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"
Sep 17 18:43:31.170042 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:43:31.170020    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14\": container with ID starting with 4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14 not found: ID does not exist" containerID="4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"
Sep 17 18:43:31.170090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.170055    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14"} err="failed to get container status \"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14\": rpc error: code = NotFound desc = could not find container \"4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14\": container with ID starting with 4eaa5fe9b893c1820995b530dd03e87b52a43fcd15ae836d21d1bb481b5ccf14 not found: ID does not exist"
Sep 17 18:43:31.170090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.170070    2577 scope.go:117] "RemoveContainer" containerID="462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"
Sep 17 18:43:31.170284 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:43:31.170267    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47\": container with ID starting with 462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47 not found: ID does not exist" containerID="462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"
Sep 17 18:43:31.170338 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:31.170291    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47"} err="failed to get container status \"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47\": rpc error: code = NotFound desc = could not find container \"462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47\": container with ID starting with 462ffc41f419d51b7f4418d165518fcda3e9072313ed0588254acc9b06985a47 not found: ID does not exist"
Sep 17 18:43:32.942133 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:32.942105    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" path="/var/lib/kubelet/pods/edb3af0c-d9c7-4847-87cc-477cea8d791b/volumes"
Sep 17 18:43:34.618863 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.618829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:34.648032 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648001    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:43:34.648347 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="storage-initializer"
Sep 17 18:43:34.648347 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="storage-initializer"
Sep 17 18:43:34.648427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kserve-container"
Sep 17 18:43:34.648427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kserve-container"
Sep 17 18:43:34.648427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kube-rbac-proxy"
Sep 17 18:43:34.648427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648379    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kube-rbac-proxy"
Sep 17 18:43:34.648557 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kube-rbac-proxy"
Sep 17 18:43:34.648557 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.648440    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb3af0c-d9c7-4847-87cc-477cea8d791b" containerName="kserve-container"
Sep 17 18:43:34.651896 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.651882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.654440 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 18:43:34.654583 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654449    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:43:34.654583 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:43:34.654714 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:43:34.654714 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654705    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:43:34.654854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:43:34.654991 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.654978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 18:43:34.664725 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.664704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:43:34.705908 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.705882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.706042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.705914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.706042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.705934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nrv\" (UniqueName: \"kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.706042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.706017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.706154 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.706071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.762841 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.762813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:34.806623 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.806593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.806744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.806646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.806744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.806667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.806744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.806692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nrv\" (UniqueName: \"kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.806744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.806738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.807186 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.807165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.807307 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.807289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.807394 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.807375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.809108 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.809089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.815375 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.815356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nrv\" (UniqueName: \"kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv\") pod \"isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:34.961779 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:34.961688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:35.235900 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:35.235876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:43:35.237731 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:43:35.237698    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b610b3c_ed3d_4b54_99b5_972b18d1beab.slice/crio-5150b33b1554f3bb0b24a15f12f0a7d9b2df1dde0020885e90d09d8738f6adb1 WatchSource:0}: Error finding container 5150b33b1554f3bb0b24a15f12f0a7d9b2df1dde0020885e90d09d8738f6adb1: Status 404 returned error can't find the container with id 5150b33b1554f3bb0b24a15f12f0a7d9b2df1dde0020885e90d09d8738f6adb1
Sep 17 18:43:36.163275 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:36.163236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerStarted","Data":"75e74e6f79f3100eaab93cf95939e2e55f47def65dfb051e3543937513c2adac"}
Sep 17 18:43:36.163687 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:36.163300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerStarted","Data":"5150b33b1554f3bb0b24a15f12f0a7d9b2df1dde0020885e90d09d8738f6adb1"}
Sep 17 18:43:40.183189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:40.183157    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerID="75e74e6f79f3100eaab93cf95939e2e55f47def65dfb051e3543937513c2adac" exitCode=0
Sep 17 18:43:40.183565 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:40.183230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerDied","Data":"75e74e6f79f3100eaab93cf95939e2e55f47def65dfb051e3543937513c2adac"}
Sep 17 18:43:41.192001 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:41.191965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerStarted","Data":"86183a23993316b85ba1513e076fd2ec8d26335ff5873bc35b8b4e37e64da7ee"}
Sep 17 18:43:43.204693 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:43.204656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerStarted","Data":"a0d16732c678958d04390d1abca3a1fc0d5d6384ffdf7da59ed8987ebbb07dee"}
Sep 17 18:43:43.205179 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:43.204738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:43.205179 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:43.204799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:43.205179 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:43.204824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerStarted","Data":"9cc00ed2d317d3919b3a164f776dcdb06d23d52d79882d8c76c775420d249a44"}
Sep 17 18:43:43.227008 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:43.226957    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podStartSLOduration=2.408420731 podStartE2EDuration="9.226942727s" totalImagesPullingTime="2.811598621s" totalInitContainerRuntime="4.006923375s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:43.224217479 +0000 UTC m=+2834.548538903" watchObservedRunningTime="2026-09-17 18:43:43.226942727 +0000 UTC m=+2834.551264150"
Sep 17 18:43:44.209190 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:44.209149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:50.225354 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:50.225320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:43:56.427772 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:43:56.427723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:10.226677 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:10.226639    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:44:14.962465 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:14.962360    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:44:20.229916 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:20.229887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:44:50.231389 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:50.231357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:44:54.668425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:54.668392    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:44:54.668864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:54.668676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" containerID="cri-o://86183a23993316b85ba1513e076fd2ec8d26335ff5873bc35b8b4e37e64da7ee" gracePeriod=30
Sep 17 18:44:54.668864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:54.668719    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-agent" containerID="cri-o://9cc00ed2d317d3919b3a164f776dcdb06d23d52d79882d8c76c775420d249a44" gracePeriod=30
Sep 17 18:44:54.668864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:54.668722    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" containerID="cri-o://a0d16732c678958d04390d1abca3a1fc0d5d6384ffdf7da59ed8987ebbb07dee" gracePeriod=30
Sep 17 18:44:54.689451 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:54.689430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:55.220689 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:55.220647    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:44:55.473614 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:55.473525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:55.529368 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:55.529335    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerID="a0d16732c678958d04390d1abca3a1fc0d5d6384ffdf7da59ed8987ebbb07dee" exitCode=2
Sep 17 18:44:55.529531 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:55.529408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerDied","Data":"a0d16732c678958d04390d1abca3a1fc0d5d6384ffdf7da59ed8987ebbb07dee"}
Sep 17 18:44:56.535477 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.535448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:56.557301 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.557273    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:44:56.560783 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.560746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.563181 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.563160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 18:44:56.563330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.563313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:44:56.563538 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.563514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:44:56.571467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.571446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:44:56.700306 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.700269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wng8\" (UniqueName: \"kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.700425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.700328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.700425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.700352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.700425 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.700393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801275 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801388 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wng8\" (UniqueName: \"kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801388 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801388 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.801973 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.801955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.803611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.803582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.808941 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.808922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wng8\" (UniqueName: \"kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:56.828710 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.828681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:56.871847 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:56.871825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:44:57.143068 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:57.143040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:44:57.144867 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:44:57.144842    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae74853c_bdc7_4267_b122_920752731f8c.slice/crio-80655155236bb209bc78eabf957602d154699b21bf5326968956afde0be90fb1 WatchSource:0}: Error finding container 80655155236bb209bc78eabf957602d154699b21bf5326968956afde0be90fb1: Status 404 returned error can't find the container with id 80655155236bb209bc78eabf957602d154699b21bf5326968956afde0be90fb1
Sep 17 18:44:57.541074 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:57.540989    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerID="86183a23993316b85ba1513e076fd2ec8d26335ff5873bc35b8b4e37e64da7ee" exitCode=0
Sep 17 18:44:57.541498 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:57.541065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerDied","Data":"86183a23993316b85ba1513e076fd2ec8d26335ff5873bc35b8b4e37e64da7ee"}
Sep 17 18:44:57.542676 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:57.542644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerStarted","Data":"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"}
Sep 17 18:44:57.542804 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:57.542705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerStarted","Data":"80655155236bb209bc78eabf957602d154699b21bf5326968956afde0be90fb1"}
Sep 17 18:44:59.849817 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.849785    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:44:59.853371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.853357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:44:59.855744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.855712    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 18:44:59.855744 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.855720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 18:44:59.863326 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.863300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:44:59.868642 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.868593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:59.928217 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.928183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:44:59.928217 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.928228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:44:59.928479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.928259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:44:59.928479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.928290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:44:59.928479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:44:59.928315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46hk\" (UniqueName: \"kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029300 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029300 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46hk\" (UniqueName: \"kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029555 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029555 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029555 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.029723 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:00.029577    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:45:00.029723 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:00.029650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls podName:d96e9dc9-fcd8-4204-897f-71fa524e58c4 nodeName:}" failed. No retries permitted until 2026-09-17 18:45:00.529629955 +0000 UTC m=+2911.853951355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-bfzn8" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:45:00.029878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.029748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.030166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.030145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.030244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.030150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.039330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.039299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46hk\" (UniqueName: \"kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.220393 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.220355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:00.226874 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.226846    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:45:00.532852 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.532770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:00.533002 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:00.532909    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:45:00.533002 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:00.532975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls podName:d96e9dc9-fcd8-4204-897f-71fa524e58c4 nodeName:}" failed. No retries permitted until 2026-09-17 18:45:01.532960213 +0000 UTC m=+2912.857281615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-bfzn8" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:45:00.640331 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:00.640293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:01.541287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:01.541254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:01.543565 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:01.543547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bfzn8\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:01.664194 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:01.664157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:01.934828 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:01.934802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:45:01.936693 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:45:01.936664    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96e9dc9_fcd8_4204_897f_71fa524e58c4.slice/crio-55c3ce3762d353c185aa1c41180f9397e37b869533dfb01736e8320d90c6c117 WatchSource:0}: Error finding container 55c3ce3762d353c185aa1c41180f9397e37b869533dfb01736e8320d90c6c117: Status 404 returned error can't find the container with id 55c3ce3762d353c185aa1c41180f9397e37b869533dfb01736e8320d90c6c117
Sep 17 18:45:02.569193 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:02.569160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerStarted","Data":"fb94017236fa87849b3e94cbde9681538bae73d0a7d412395ce443e4133c6c5d"}
Sep 17 18:45:02.569667 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:02.569225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerStarted","Data":"55c3ce3762d353c185aa1c41180f9397e37b869533dfb01736e8320d90c6c117"}
Sep 17 18:45:02.570923 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:02.570907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/0.log"
Sep 17 18:45:02.571004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:02.570937    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae74853c-bdc7-4267-b122-920752731f8c" containerID="7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7" exitCode=1
Sep 17 18:45:02.571004 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:02.570971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerDied","Data":"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"}
Sep 17 18:45:03.577076 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:03.577052    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/0.log"
Sep 17 18:45:03.577467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:03.577163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerStarted","Data":"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"}
Sep 17 18:45:05.220078 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:05.220040    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:05.220479 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:05.220159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:45:06.530111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:06.530068    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:45:06.530614 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:06.530266    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer" containerID="cri-o://cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae" gracePeriod=30
Sep 17 18:45:06.550386 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:06.550329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:07.599498 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:07.599459    2577 generic.go:356] "Generic (PLEG): container finished" podID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerID="fb94017236fa87849b3e94cbde9681538bae73d0a7d412395ce443e4133c6c5d" exitCode=0
Sep 17 18:45:07.599916 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:07.599533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerDied","Data":"fb94017236fa87849b3e94cbde9681538bae73d0a7d412395ce443e4133c6c5d"}
Sep 17 18:45:07.600727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:07.600712    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:45:07.655115 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:07.655077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:08.088722 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:08.088635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:09.888494 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:09.888467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/1.log"
Sep 17 18:45:09.888952 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:09.888931    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/0.log"
Sep 17 18:45:09.891334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:09.891314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:45:10.021606 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.021524    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls\") pod \"ae74853c-bdc7-4267-b122-920752731f8c\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") "
Sep 17 18:45:10.021822 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.021630    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location\") pod \"ae74853c-bdc7-4267-b122-920752731f8c\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") "
Sep 17 18:45:10.021822 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.021657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wng8\" (UniqueName: \"kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8\") pod \"ae74853c-bdc7-4267-b122-920752731f8c\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") "
Sep 17 18:45:10.021822 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.021697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config\") pod \"ae74853c-bdc7-4267-b122-920752731f8c\" (UID: \"ae74853c-bdc7-4267-b122-920752731f8c\") "
Sep 17 18:45:10.022012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.021948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae74853c-bdc7-4267-b122-920752731f8c" (UID: "ae74853c-bdc7-4267-b122-920752731f8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:10.022195 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.022157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae74853c-bdc7-4267-b122-920752731f8c" (UID: "ae74853c-bdc7-4267-b122-920752731f8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:10.023967 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.023941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae74853c-bdc7-4267-b122-920752731f8c" (UID: "ae74853c-bdc7-4267-b122-920752731f8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:10.024058 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.023977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8" (OuterVolumeSpecName: "kube-api-access-5wng8") pod "ae74853c-bdc7-4267-b122-920752731f8c" (UID: "ae74853c-bdc7-4267-b122-920752731f8c"). InnerVolumeSpecName "kube-api-access-5wng8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:10.122478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.122447    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wng8\" (UniqueName: \"kubernetes.io/projected/ae74853c-bdc7-4267-b122-920752731f8c-kube-api-access-5wng8\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:10.122478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.122472    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae74853c-bdc7-4267-b122-920752731f8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:10.122478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.122482    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae74853c-bdc7-4267-b122-920752731f8c-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:10.122791 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.122496    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae74853c-bdc7-4267-b122-920752731f8c-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:10.219958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.219916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:10.226541 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.226500    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:45:10.621176 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621149    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/1.log"
Sep 17 18:45:10.621643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621623    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr_ae74853c-bdc7-4267-b122-920752731f8c/storage-initializer/0.log"
Sep 17 18:45:10.621746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621666    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae74853c-bdc7-4267-b122-920752731f8c" containerID="cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae" exitCode=1
Sep 17 18:45:10.621746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"
Sep 17 18:45:10.621871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerDied","Data":"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"}
Sep 17 18:45:10.621871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr" event={"ID":"ae74853c-bdc7-4267-b122-920752731f8c","Type":"ContainerDied","Data":"80655155236bb209bc78eabf957602d154699b21bf5326968956afde0be90fb1"}
Sep 17 18:45:10.621871 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.621859    2577 scope.go:117] "RemoveContainer" containerID="cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"
Sep 17 18:45:10.632919 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.632899    2577 scope.go:117] "RemoveContainer" containerID="7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"
Sep 17 18:45:10.642547 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.642524    2577 scope.go:117] "RemoveContainer" containerID="cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"
Sep 17 18:45:10.642882 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:10.642861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae\": container with ID starting with cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae not found: ID does not exist" containerID="cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"
Sep 17 18:45:10.642986 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.642891    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae"} err="failed to get container status \"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae\": rpc error: code = NotFound desc = could not find container \"cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae\": container with ID starting with cad0f6190f7a8e9d73cd2bf89fd84a1fd36e2bcb5a6a41eb01e25f3ed9db44ae not found: ID does not exist"
Sep 17 18:45:10.642986 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.642908    2577 scope.go:117] "RemoveContainer" containerID="7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"
Sep 17 18:45:10.643207 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:45:10.643186    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7\": container with ID starting with 7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7 not found: ID does not exist" containerID="7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"
Sep 17 18:45:10.643261 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.643213    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7"} err="failed to get container status \"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7\": rpc error: code = NotFound desc = could not find container \"7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7\": container with ID starting with 7d5d153666aefc11d62789f4987bec723c95b4f3b8995092ad1a13c328de9ad7 not found: ID does not exist"
Sep 17 18:45:10.661415 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.661370    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:45:10.664608 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.664567    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6759c5f8df-mbvpr"]
Sep 17 18:45:10.943239 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:10.943159    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae74853c-bdc7-4267-b122-920752731f8c" path="/var/lib/kubelet/pods/ae74853c-bdc7-4267-b122-920752731f8c/volumes"
Sep 17 18:45:15.220810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:15.220747    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:18.667347 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.667324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerStarted","Data":"e231152800fe3669ca78e65758b5dbf68c0ae7f21a2c329767ee8c2585f59700"}
Sep 17 18:45:18.667699 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.667378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerStarted","Data":"4edc09f409d5b22f7af9370083fe5f9d0b14a51a5903536e00c564414f721eb4"}
Sep 17 18:45:18.667699 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.667631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:18.667814 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.667792    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:18.668958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.668935    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:45:18.687428 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:18.687377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podStartSLOduration=3.945492525 podStartE2EDuration="19.687360458s" totalImagesPullingTime="10.937303899s" totalInitContainerRuntime="4.804564034s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:18.684957569 +0000 UTC m=+2930.009278992" watchObservedRunningTime="2026-09-17 18:45:18.687360458 +0000 UTC m=+2930.011681881"
Sep 17 18:45:19.672230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:19.672193    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:45:20.220152 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:20.220113    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:20.226383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:20.226353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:45:20.226497 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:20.226482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:45:24.677569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:24.677544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:45:24.678383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:24.678104    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:45:24.699266 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:24.699241    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerID="9cc00ed2d317d3919b3a164f776dcdb06d23d52d79882d8c76c775420d249a44" exitCode=137
Sep 17 18:45:24.699399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:24.699308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerDied","Data":"9cc00ed2d317d3919b3a164f776dcdb06d23d52d79882d8c76c775420d249a44"}
Sep 17 18:45:25.220523 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.220484    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:45:25.319030 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.319008    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:45:25.350941 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.350910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67nrv\" (UniqueName: \"kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv\") pod \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") "
Sep 17 18:45:25.351093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config\") pod \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") "
Sep 17 18:45:25.351093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location\") pod \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") "
Sep 17 18:45:25.351214 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert\") pod \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") "
Sep 17 18:45:25.351214 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351146    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls\") pod \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\" (UID: \"2b610b3c-ed3d-4b54-99b5-972b18d1beab\") "
Sep 17 18:45:25.351426 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b610b3c-ed3d-4b54-99b5-972b18d1beab" (UID: "2b610b3c-ed3d-4b54-99b5-972b18d1beab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:25.351507 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b610b3c-ed3d-4b54-99b5-972b18d1beab" (UID: "2b610b3c-ed3d-4b54-99b5-972b18d1beab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:25.351507 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.351468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b610b3c-ed3d-4b54-99b5-972b18d1beab" (UID: "2b610b3c-ed3d-4b54-99b5-972b18d1beab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:25.352994 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.352971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv" (OuterVolumeSpecName: "kube-api-access-67nrv") pod "2b610b3c-ed3d-4b54-99b5-972b18d1beab" (UID: "2b610b3c-ed3d-4b54-99b5-972b18d1beab"). InnerVolumeSpecName "kube-api-access-67nrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:25.353283 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.353259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b610b3c-ed3d-4b54-99b5-972b18d1beab" (UID: "2b610b3c-ed3d-4b54-99b5-972b18d1beab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:25.451810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.451714    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:25.451810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.451742    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:25.451810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.451772    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b610b3c-ed3d-4b54-99b5-972b18d1beab-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:25.451810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.451787    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b610b3c-ed3d-4b54-99b5-972b18d1beab-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:25.451810 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.451799    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67nrv\" (UniqueName: \"kubernetes.io/projected/2b610b3c-ed3d-4b54-99b5-972b18d1beab-kube-api-access-67nrv\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:45:25.706168 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.706075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9" event={"ID":"2b610b3c-ed3d-4b54-99b5-972b18d1beab","Type":"ContainerDied","Data":"5150b33b1554f3bb0b24a15f12f0a7d9b2df1dde0020885e90d09d8738f6adb1"}
Sep 17 18:45:25.706168 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.706119    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"
Sep 17 18:45:25.706168 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.706154    2577 scope.go:117] "RemoveContainer" containerID="a0d16732c678958d04390d1abca3a1fc0d5d6384ffdf7da59ed8987ebbb07dee"
Sep 17 18:45:25.714923 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.714908    2577 scope.go:117] "RemoveContainer" containerID="9cc00ed2d317d3919b3a164f776dcdb06d23d52d79882d8c76c775420d249a44"
Sep 17 18:45:25.722366 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.722348    2577 scope.go:117] "RemoveContainer" containerID="86183a23993316b85ba1513e076fd2ec8d26335ff5873bc35b8b4e37e64da7ee"
Sep 17 18:45:25.727704 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.727685    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:45:25.730219 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.730199    2577 scope.go:117] "RemoveContainer" containerID="75e74e6f79f3100eaab93cf95939e2e55f47def65dfb051e3543937513c2adac"
Sep 17 18:45:25.731413 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:25.731391    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cb98679d4-6kzt9"]
Sep 17 18:45:26.941908 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:26.941875    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" path="/var/lib/kubelet/pods/2b610b3c-ed3d-4b54-99b5-972b18d1beab/volumes"
Sep 17 18:45:34.678767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:34.678713    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:45:44.679026 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:44.678941    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:45:54.678870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:45:54.678824    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 18:46:04.678767 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:04.678719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:46:11.191819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.191783    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:46:11.192293 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.192085    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container" containerID="cri-o://4edc09f409d5b22f7af9370083fe5f9d0b14a51a5903536e00c564414f721eb4" gracePeriod=30
Sep 17 18:46:11.192293 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.192157    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kube-rbac-proxy" containerID="cri-o://e231152800fe3669ca78e65758b5dbf68c0ae7f21a2c329767ee8c2585f59700" gracePeriod=30
Sep 17 18:46:11.216786 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.216738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:11.925683 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.925646    2577 generic.go:356] "Generic (PLEG): container finished" podID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerID="e231152800fe3669ca78e65758b5dbf68c0ae7f21a2c329767ee8c2585f59700" exitCode=2
Sep 17 18:46:11.925864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:11.925695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerDied","Data":"e231152800fe3669ca78e65758b5dbf68c0ae7f21a2c329767ee8c2585f59700"}
Sep 17 18:46:13.938369 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:13.938340    2577 generic.go:356] "Generic (PLEG): container finished" podID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerID="4edc09f409d5b22f7af9370083fe5f9d0b14a51a5903536e00c564414f721eb4" exitCode=0
Sep 17 18:46:13.938783 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:13.938412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerDied","Data":"4edc09f409d5b22f7af9370083fe5f9d0b14a51a5903536e00c564414f721eb4"}
Sep 17 18:46:14.030779 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.030741    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:46:14.162446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location\") pod \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") "
Sep 17 18:46:14.162446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert\") pod \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") "
Sep 17 18:46:14.162446 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b46hk\" (UniqueName: \"kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk\") pod \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") "
Sep 17 18:46:14.162728 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162547    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config\") pod \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") "
Sep 17 18:46:14.162728 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") pod \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\" (UID: \"d96e9dc9-fcd8-4204-897f-71fa524e58c4\") "
Sep 17 18:46:14.162882 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d96e9dc9-fcd8-4204-897f-71fa524e58c4" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:14.162977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.162957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d96e9dc9-fcd8-4204-897f-71fa524e58c4" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:14.164643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.164620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk" (OuterVolumeSpecName: "kube-api-access-b46hk") pod "d96e9dc9-fcd8-4204-897f-71fa524e58c4" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4"). InnerVolumeSpecName "kube-api-access-b46hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:14.164643 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.164640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d96e9dc9-fcd8-4204-897f-71fa524e58c4" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:14.174720 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.174699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d96e9dc9-fcd8-4204-897f-71fa524e58c4" (UID: "d96e9dc9-fcd8-4204-897f-71fa524e58c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:14.263835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.263804    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:14.263835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.263836    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d96e9dc9-fcd8-4204-897f-71fa524e58c4-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:14.264031 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.263852    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:14.264031 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.263870    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d96e9dc9-fcd8-4204-897f-71fa524e58c4-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:14.264031 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.263882    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b46hk\" (UniqueName: \"kubernetes.io/projected/d96e9dc9-fcd8-4204-897f-71fa524e58c4-kube-api-access-b46hk\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:14.945203 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.945172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"
Sep 17 18:46:14.945694 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.945200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8" event={"ID":"d96e9dc9-fcd8-4204-897f-71fa524e58c4","Type":"ContainerDied","Data":"55c3ce3762d353c185aa1c41180f9397e37b869533dfb01736e8320d90c6c117"}
Sep 17 18:46:14.945694 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.945265    2577 scope.go:117] "RemoveContainer" containerID="e231152800fe3669ca78e65758b5dbf68c0ae7f21a2c329767ee8c2585f59700"
Sep 17 18:46:14.954040 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.953950    2577 scope.go:117] "RemoveContainer" containerID="4edc09f409d5b22f7af9370083fe5f9d0b14a51a5903536e00c564414f721eb4"
Sep 17 18:46:14.962026 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.962000    2577 scope.go:117] "RemoveContainer" containerID="fb94017236fa87849b3e94cbde9681538bae73d0a7d412395ce443e4133c6c5d"
Sep 17 18:46:14.969021 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.968997    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:46:14.971747 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:14.971728    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bfzn8"]
Sep 17 18:46:16.352485 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.352455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:16.380673 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.380645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:46:16.380983 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.380971    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.380985    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381000    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381007    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381019    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="storage-initializer"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381024    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="storage-initializer"
Sep 17 18:46:16.381035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381031    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381037    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381046    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381051    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381059    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-agent"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381064    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-agent"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381070    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381075    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381083    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381088    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381094    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381099    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381145    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381152    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kserve-container"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381157    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-agent"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381165    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381173    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kserve-container"
Sep 17 18:46:16.381224 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381177    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b610b3c-ed3d-4b54-99b5-972b18d1beab" containerName="kube-rbac-proxy"
Sep 17 18:46:16.381716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.381273    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae74853c-bdc7-4267-b122-920752731f8c" containerName="storage-initializer"
Sep 17 18:46:16.386912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.384909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.388378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.387822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:46:16.388378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.387921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:46:16.388378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.387958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 18:46:16.388378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.388332    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:46:16.388931 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.388911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:46:16.389189 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.389171    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:46:16.389268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.389174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:46:16.393799 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.393777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:46:16.452989 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.452955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:16.487889 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.487861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.488029 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.487909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.488029 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.488000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxllf\" (UniqueName: \"kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.488144 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.488025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.488144 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.488071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.588679 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.588643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.588885 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.588695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.588885 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.588740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.588885 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.588831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxllf\" (UniqueName: \"kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.589053 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.588934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.589258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.589239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.589473 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.589451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.589547 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.589532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.591258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.591231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.596967 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.596944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxllf\" (UniqueName: \"kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-97w74\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.699610 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.699584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:16.941501 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.941467    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96e9dc9-fcd8-4204-897f-71fa524e58c4" path="/var/lib/kubelet/pods/d96e9dc9-fcd8-4204-897f-71fa524e58c4/volumes"
Sep 17 18:46:16.968625 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:16.968596    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:46:16.973214 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:46:16.973183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc067b744_6fa1_4ab9_83bf_c1deea540e62.slice/crio-1b3733195b076bf1336d488206091bc549bca5cddcb7a7e20739fe8b8381df07 WatchSource:0}: Error finding container 1b3733195b076bf1336d488206091bc549bca5cddcb7a7e20739fe8b8381df07: Status 404 returned error can't find the container with id 1b3733195b076bf1336d488206091bc549bca5cddcb7a7e20739fe8b8381df07
Sep 17 18:46:17.961930 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:17.961896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerStarted","Data":"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"}
Sep 17 18:46:17.962319 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:17.961957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerStarted","Data":"1b3733195b076bf1336d488206091bc549bca5cddcb7a7e20739fe8b8381df07"}
Sep 17 18:46:22.985310 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:22.985221    2577 generic.go:356] "Generic (PLEG): container finished" podID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerID="ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a" exitCode=0
Sep 17 18:46:22.985310 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:22.985261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerDied","Data":"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"}
Sep 17 18:46:23.991518 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:23.991482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerStarted","Data":"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"}
Sep 17 18:46:23.991935 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:23.991549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerStarted","Data":"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"}
Sep 17 18:46:23.991935 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:23.991767    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:24.012127 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:24.012079    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podStartSLOduration=2.344826846 podStartE2EDuration="8.012065317s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.667238471s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:24.010200638 +0000 UTC m=+2995.334522063" watchObservedRunningTime="2026-09-17 18:46:24.012065317 +0000 UTC m=+2995.336386739"
Sep 17 18:46:24.995719 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:24.995682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:24.997058 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:24.997027    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:46:25.376011 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:25.375979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:25.999560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:25.999522    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:46:31.004093 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:31.004060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:46:31.004549 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:31.004524    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:46:37.725946 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:37.725912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:38.779772 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.779718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:38.827244 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.827211    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:38.830854 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.830838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.833246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.833225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 18:46:38.833356 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.833274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:46:38.833510 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.833497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:46:38.841022 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.841002    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:38.875844 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.875813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.875993 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.875866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhwr\" (UniqueName: \"kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.875993 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.875901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.875993 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.875933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.976950 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.976912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhwr\" (UniqueName: \"kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.977134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.976956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.977134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.976993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.977134 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.977064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.977369 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:46:38.977135    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:46:38.977369 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:46:38.977219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls podName:23b8eb46-4fa2-4d38-98b1-c016424da7f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:46:39.47719797 +0000 UTC m=+3010.801519388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" (UID: "23b8eb46-4fa2-4d38-98b1-c016424da7f5") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:46:38.977571 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.977552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.977779 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.977743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:38.985330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:38.985305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhwr\" (UniqueName: \"kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:39.132981 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:39.132950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:39.482046 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:39.481940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:39.484381 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:39.484360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:39.740851 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:39.740772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:40.016669 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:40.016644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:40.018743 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:46:40.018720    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b8eb46_4fa2_4d38_98b1_c016424da7f5.slice/crio-74a8b557f28a2df6105d8efce78b138f37c19c44ff76f5e99356a0f5f88d1815 WatchSource:0}: Error finding container 74a8b557f28a2df6105d8efce78b138f37c19c44ff76f5e99356a0f5f88d1815: Status 404 returned error can't find the container with id 74a8b557f28a2df6105d8efce78b138f37c19c44ff76f5e99356a0f5f88d1815
Sep 17 18:46:40.066112 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:40.066081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerStarted","Data":"74a8b557f28a2df6105d8efce78b138f37c19c44ff76f5e99356a0f5f88d1815"}
Sep 17 18:46:41.004567 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:41.004528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:46:41.072600 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:41.072568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerStarted","Data":"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"}
Sep 17 18:46:43.084656 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:43.084624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/0.log"
Sep 17 18:46:43.085103 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:43.084664    2577 generic.go:356] "Generic (PLEG): container finished" podID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerID="8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59" exitCode=1
Sep 17 18:46:43.085103 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:43.084722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerDied","Data":"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"}
Sep 17 18:46:44.094092 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:44.094069    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/0.log"
Sep 17 18:46:44.094478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:44.094182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerStarted","Data":"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"}
Sep 17 18:46:48.803438 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.803399    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:48.803859 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.803676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer" containerID="cri-o://a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f" gracePeriod=30
Sep 17 18:46:48.820724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.820680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:48.929220 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.929199    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/1.log"
Sep 17 18:46:48.929641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.929623    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/0.log"
Sep 17 18:46:48.931857 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:48.931841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:49.067987 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.067898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkhwr\" (UniqueName: \"kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr\") pod \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") "
Sep 17 18:46:49.068150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.067992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location\") pod \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") "
Sep 17 18:46:49.068150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.068062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") pod \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") "
Sep 17 18:46:49.068150 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.068111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config\") pod \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\" (UID: \"23b8eb46-4fa2-4d38-98b1-c016424da7f5\") "
Sep 17 18:46:49.068290 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.068261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23b8eb46-4fa2-4d38-98b1-c016424da7f5" (UID: "23b8eb46-4fa2-4d38-98b1-c016424da7f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:49.068444 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.068424    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23b8eb46-4fa2-4d38-98b1-c016424da7f5" (UID: "23b8eb46-4fa2-4d38-98b1-c016424da7f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:49.068512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.068434    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:49.070062 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.070031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr" (OuterVolumeSpecName: "kube-api-access-dkhwr") pod "23b8eb46-4fa2-4d38-98b1-c016424da7f5" (UID: "23b8eb46-4fa2-4d38-98b1-c016424da7f5"). InnerVolumeSpecName "kube-api-access-dkhwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:49.070174 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.070133    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23b8eb46-4fa2-4d38-98b1-c016424da7f5" (UID: "23b8eb46-4fa2-4d38-98b1-c016424da7f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:49.118339 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118306    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/1.log"
Sep 17 18:46:49.118717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118700    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj_23b8eb46-4fa2-4d38-98b1-c016424da7f5/storage-initializer/0.log"
Sep 17 18:46:49.118842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118736    2577 generic.go:356] "Generic (PLEG): container finished" podID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerID="a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f" exitCode=1
Sep 17 18:46:49.118842 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"
Sep 17 18:46:49.118967 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerDied","Data":"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"}
Sep 17 18:46:49.118967 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj" event={"ID":"23b8eb46-4fa2-4d38-98b1-c016424da7f5","Type":"ContainerDied","Data":"74a8b557f28a2df6105d8efce78b138f37c19c44ff76f5e99356a0f5f88d1815"}
Sep 17 18:46:49.119077 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.118986    2577 scope.go:117] "RemoveContainer" containerID="a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"
Sep 17 18:46:49.127569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.127550    2577 scope.go:117] "RemoveContainer" containerID="8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"
Sep 17 18:46:49.134916 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.134894    2577 scope.go:117] "RemoveContainer" containerID="a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"
Sep 17 18:46:49.135189 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:46:49.135168    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f\": container with ID starting with a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f not found: ID does not exist" containerID="a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"
Sep 17 18:46:49.135250 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.135198    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f"} err="failed to get container status \"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f\": rpc error: code = NotFound desc = could not find container \"a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f\": container with ID starting with a1a554f315cadad511482b0166cb9a7a49b996a46fbd616506c6516e69ba213f not found: ID does not exist"
Sep 17 18:46:49.135250 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.135216    2577 scope.go:117] "RemoveContainer" containerID="8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"
Sep 17 18:46:49.135443 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:46:49.135429    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59\": container with ID starting with 8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59 not found: ID does not exist" containerID="8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"
Sep 17 18:46:49.135490 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.135446    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59"} err="failed to get container status \"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59\": rpc error: code = NotFound desc = could not find container \"8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59\": container with ID starting with 8cf74e578d53a1c0d7c3b1485f941a3f3e90e1714a63cef6b993e3cf563bef59 not found: ID does not exist"
Sep 17 18:46:49.157049 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.157026    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:49.160882 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.160858    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58495fdf8f-8mzrj"]
Sep 17 18:46:49.169685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.169666    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b8eb46-4fa2-4d38-98b1-c016424da7f5-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:49.169685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.169685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:49.169821 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.169695    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhwr\" (UniqueName: \"kubernetes.io/projected/23b8eb46-4fa2-4d38-98b1-c016424da7f5-kube-api-access-dkhwr\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:46:49.902447 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:49.902421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:50.218611 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:50.218508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:50.942173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:50.942144    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" path="/var/lib/kubelet/pods/23b8eb46-4fa2-4d38-98b1-c016424da7f5/volumes"
Sep 17 18:46:51.004895 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:51.004859    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:46:55.723476 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:46:55.723412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:01.005212 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:01.005171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 18:47:07.515843 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:07.515802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:11.004958 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:11.004921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:47:17.795399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:17.795363    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:47:17.797870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:17.795599    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container" containerID="cri-o://3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa" gracePeriod=30
Sep 17 18:47:17.797870 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:17.795638    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kube-rbac-proxy" containerID="cri-o://b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50" gracePeriod=30
Sep 17 18:47:17.807240 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:17.807219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:18.260547 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:18.260516    2577 generic.go:356] "Generic (PLEG): container finished" podID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerID="b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50" exitCode=2
Sep 17 18:47:18.260720 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:18.260571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerDied","Data":"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"}
Sep 17 18:47:20.637015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.636992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:47:20.751878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.751788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert\") pod \"c067b744-6fa1-4ab9-83bf-c1deea540e62\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") "
Sep 17 18:47:20.751878 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.751853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxllf\" (UniqueName: \"kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf\") pod \"c067b744-6fa1-4ab9-83bf-c1deea540e62\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") "
Sep 17 18:47:20.752064 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.751952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config\") pod \"c067b744-6fa1-4ab9-83bf-c1deea540e62\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") "
Sep 17 18:47:20.752064 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls\") pod \"c067b744-6fa1-4ab9-83bf-c1deea540e62\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") "
Sep 17 18:47:20.752149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location\") pod \"c067b744-6fa1-4ab9-83bf-c1deea540e62\" (UID: \"c067b744-6fa1-4ab9-83bf-c1deea540e62\") "
Sep 17 18:47:20.752287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c067b744-6fa1-4ab9-83bf-c1deea540e62" (UID: "c067b744-6fa1-4ab9-83bf-c1deea540e62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:20.752432 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c067b744-6fa1-4ab9-83bf-c1deea540e62" (UID: "c067b744-6fa1-4ab9-83bf-c1deea540e62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:20.752488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752444    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:47:20.752488 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.752459    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:47:20.754071 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.754050    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c067b744-6fa1-4ab9-83bf-c1deea540e62" (UID: "c067b744-6fa1-4ab9-83bf-c1deea540e62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:20.754371 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.754354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf" (OuterVolumeSpecName: "kube-api-access-cxllf") pod "c067b744-6fa1-4ab9-83bf-c1deea540e62" (UID: "c067b744-6fa1-4ab9-83bf-c1deea540e62"). InnerVolumeSpecName "kube-api-access-cxllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:20.761294 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.761273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c067b744-6fa1-4ab9-83bf-c1deea540e62" (UID: "c067b744-6fa1-4ab9-83bf-c1deea540e62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:20.853525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.853495    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c067b744-6fa1-4ab9-83bf-c1deea540e62-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:47:20.853525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.853521    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c067b744-6fa1-4ab9-83bf-c1deea540e62-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:47:20.853525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:20.853531    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxllf\" (UniqueName: \"kubernetes.io/projected/c067b744-6fa1-4ab9-83bf-c1deea540e62-kube-api-access-cxllf\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:47:21.274920 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.274883    2577 generic.go:356] "Generic (PLEG): container finished" podID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerID="3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa" exitCode=0
Sep 17 18:47:21.275090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.274951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"
Sep 17 18:47:21.275090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.274968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerDied","Data":"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"}
Sep 17 18:47:21.275090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.275030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74" event={"ID":"c067b744-6fa1-4ab9-83bf-c1deea540e62","Type":"ContainerDied","Data":"1b3733195b076bf1336d488206091bc549bca5cddcb7a7e20739fe8b8381df07"}
Sep 17 18:47:21.275090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.275057    2577 scope.go:117] "RemoveContainer" containerID="b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"
Sep 17 18:47:21.283154 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.283134    2577 scope.go:117] "RemoveContainer" containerID="3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"
Sep 17 18:47:21.290516 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.290499    2577 scope.go:117] "RemoveContainer" containerID="ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"
Sep 17 18:47:21.294427 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.294407    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:47:21.297724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.297704    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-97w74"]
Sep 17 18:47:21.298299 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.298283    2577 scope.go:117] "RemoveContainer" containerID="b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"
Sep 17 18:47:21.298568 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:47:21.298550    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50\": container with ID starting with b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50 not found: ID does not exist" containerID="b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"
Sep 17 18:47:21.298622 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.298584    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50"} err="failed to get container status \"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50\": rpc error: code = NotFound desc = could not find container \"b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50\": container with ID starting with b208e82a5672c4c0afa8fdbac05932ad599f84e548f3b18a16922828960bbd50 not found: ID does not exist"
Sep 17 18:47:21.298622 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.298602    2577 scope.go:117] "RemoveContainer" containerID="3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"
Sep 17 18:47:21.298914 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:47:21.298895    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa\": container with ID starting with 3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa not found: ID does not exist" containerID="3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"
Sep 17 18:47:21.298979 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.298918    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa"} err="failed to get container status \"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa\": rpc error: code = NotFound desc = could not find container \"3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa\": container with ID starting with 3cf8b2aa076869823f3633d27c54fa5702507a1e60ec00b579401ef69d0876aa not found: ID does not exist"
Sep 17 18:47:21.298979 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.298933    2577 scope.go:117] "RemoveContainer" containerID="ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"
Sep 17 18:47:21.299175 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:47:21.299158    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a\": container with ID starting with ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a not found: ID does not exist" containerID="ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"
Sep 17 18:47:21.299216 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:21.299180    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a"} err="failed to get container status \"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a\": rpc error: code = NotFound desc = could not find container \"ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a\": container with ID starting with ec9db6167732ac19b051c11fbf114278c8dcc3e8e158cbb436df8d39f841d74a not found: ID does not exist"
Sep 17 18:47:22.942166 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:22.942134    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" path="/var/lib/kubelet/pods/c067b744-6fa1-4ab9-83bf-c1deea540e62/volumes"
Sep 17 18:47:22.960697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:22.960657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:22.999995 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:22.999970    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:47:23.000287 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kube-rbac-proxy"
Sep 17 18:47:23.000330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kube-rbac-proxy"
Sep 17 18:47:23.000330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000319    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container"
Sep 17 18:47:23.000330 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000324    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000341    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000356    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000402    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000411    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kube-rbac-proxy"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000417    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b8eb46-4fa2-4d38-98b1-c016424da7f5" containerName="storage-initializer"
Sep 17 18:47:23.000486 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.000425    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c067b744-6fa1-4ab9-83bf-c1deea540e62" containerName="kserve-container"
Sep 17 18:47:23.004886 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.004871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.007267 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:47:23.007267 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007258    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:47:23.007459 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 18:47:23.007459 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:47:23.007459 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:47:23.007459 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.007305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:47:23.008177 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.008156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:47:23.013771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.013733    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:47:23.073236 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.073202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.073412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.073258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8f2\" (UniqueName: \"kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.073412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.073319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.073412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.073339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.073412 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.073359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174628 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.174594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8f2\" (UniqueName: \"kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174832 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.174646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174832 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.174678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174832 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.174712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174832 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.174749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.174832 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:47:23.174781    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 18:47:23.175119 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:47:23.174846    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls podName:91eafde7-cda9-423a-b104-54c1024ad053 nodeName:}" failed. No retries permitted until 2026-09-17 18:47:23.674826185 +0000 UTC m=+3054.999147609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" (UID: "91eafde7-cda9-423a-b104-54c1024ad053") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 18:47:23.175232 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.175214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.175433 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.175414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.175524 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.175504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.183406 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.183386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8f2\" (UniqueName: \"kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.531158 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.531111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:23.678407 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.678367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.680764 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.680735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:23.915935 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:23.915903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:24.187506 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:24.187483    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:47:24.189499 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:47:24.189469    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eafde7_cda9_423a_b104_54c1024ad053.slice/crio-04af65eee948e95198e8c893c43ddcfe89bcf01cf39ec127cb76536451e45eed WatchSource:0}: Error finding container 04af65eee948e95198e8c893c43ddcfe89bcf01cf39ec127cb76536451e45eed: Status 404 returned error can't find the container with id 04af65eee948e95198e8c893c43ddcfe89bcf01cf39ec127cb76536451e45eed
Sep 17 18:47:24.289891 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:24.289858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerStarted","Data":"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"}
Sep 17 18:47:24.290032 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:24.289929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerStarted","Data":"04af65eee948e95198e8c893c43ddcfe89bcf01cf39ec127cb76536451e45eed"}
Sep 17 18:47:29.319523 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:29.319486    2577 generic.go:356] "Generic (PLEG): container finished" podID="91eafde7-cda9-423a-b104-54c1024ad053" containerID="16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6" exitCode=0
Sep 17 18:47:29.319924 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:29.319560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerDied","Data":"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"}
Sep 17 18:47:30.326042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.326009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerStarted","Data":"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"}
Sep 17 18:47:30.326519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.326069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerStarted","Data":"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"}
Sep 17 18:47:30.326519 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.326382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:30.326640 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.326528    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:30.327856 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.327834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:47:30.351325 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:30.351279    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podStartSLOduration=4.151542907 podStartE2EDuration="8.351266182s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.199723275s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:30.349124789 +0000 UTC m=+3061.673446214" watchObservedRunningTime="2026-09-17 18:47:30.351266182 +0000 UTC m=+3061.675587604"
Sep 17 18:47:31.331102 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:31.331057    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:47:32.334876 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:32.334835    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:47:37.339727 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:37.339692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:47:37.340334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:37.340309    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:47:47.340659 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:47.340620    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:47:57.340658 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:47:57.340617    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:48:07.340734 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:07.340691    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:48:09.951528 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:09.951499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:11.021697 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.021659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:11.026982 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.026958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:11.029456 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.029443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.031746 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.031719    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:48:11.031873 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.031725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:48:11.031873 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.031780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 18:48:11.040372 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.040347    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:11.072231 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.072211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.072352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.072277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxz6\" (UniqueName: \"kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.072352 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.072310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.072475 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.072365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173138 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxz6\" (UniqueName: \"kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173627 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.173965 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.173943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.175625 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.175606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.181487 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.181462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxz6\" (UniqueName: \"kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.219082 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.219051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:11.340012 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.339932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:11.615873 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:11.615848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:11.617920 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:48:11.617892    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a808197_05be_40f4_a97d_1f8ab741e991.slice/crio-4ae84dfcfe48562a23d3dcc37f1dd1f312270b3bcc4aa53b7fa8b0a5d9f7f892 WatchSource:0}: Error finding container 4ae84dfcfe48562a23d3dcc37f1dd1f312270b3bcc4aa53b7fa8b0a5d9f7f892: Status 404 returned error can't find the container with id 4ae84dfcfe48562a23d3dcc37f1dd1f312270b3bcc4aa53b7fa8b0a5d9f7f892
Sep 17 18:48:12.511523 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:12.511482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerStarted","Data":"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"}
Sep 17 18:48:12.511977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:12.511545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerStarted","Data":"4ae84dfcfe48562a23d3dcc37f1dd1f312270b3bcc4aa53b7fa8b0a5d9f7f892"}
Sep 17 18:48:16.538433 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:16.538406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/0.log"
Sep 17 18:48:16.538819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:16.538446    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a808197-05be-40f4-a97d-1f8ab741e991" containerID="ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f" exitCode=1
Sep 17 18:48:16.538819 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:16.538479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerDied","Data":"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"}
Sep 17 18:48:17.341484 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:17.341450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:48:17.545075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:17.545046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/0.log"
Sep 17 18:48:17.545442 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:17.545162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerStarted","Data":"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"}
Sep 17 18:48:20.994087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:20.994057    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:20.994456 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:20.994249    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer" containerID="cri-o://f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5" gracePeriod=30
Sep 17 18:48:21.002515 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:21.002466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:22.135097 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.135068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:22.138355 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.138337    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/1.log"
Sep 17 18:48:22.138724 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.138709    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/0.log"
Sep 17 18:48:22.139961 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.139946    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:22.268698 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.268605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxz6\" (UniqueName: \"kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6\") pod \"9a808197-05be-40f4-a97d-1f8ab741e991\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") "
Sep 17 18:48:22.268892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.268697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config\") pod \"9a808197-05be-40f4-a97d-1f8ab741e991\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") "
Sep 17 18:48:22.268892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.268726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls\") pod \"9a808197-05be-40f4-a97d-1f8ab741e991\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") "
Sep 17 18:48:22.268892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.268794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location\") pod \"9a808197-05be-40f4-a97d-1f8ab741e991\" (UID: \"9a808197-05be-40f4-a97d-1f8ab741e991\") "
Sep 17 18:48:22.269158 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.269131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a808197-05be-40f4-a97d-1f8ab741e991" (UID: "9a808197-05be-40f4-a97d-1f8ab741e991"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:22.269238 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.269135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a808197-05be-40f4-a97d-1f8ab741e991" (UID: "9a808197-05be-40f4-a97d-1f8ab741e991"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:48:22.269525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.269506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:22.270835 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.270812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a808197-05be-40f4-a97d-1f8ab741e991" (UID: "9a808197-05be-40f4-a97d-1f8ab741e991"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:22.270938 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.270892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6" (OuterVolumeSpecName: "kube-api-access-vhxz6") pod "9a808197-05be-40f4-a97d-1f8ab741e991" (UID: "9a808197-05be-40f4-a97d-1f8ab741e991"). InnerVolumeSpecName "kube-api-access-vhxz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:22.369617 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.369591    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxz6\" (UniqueName: \"kubernetes.io/projected/9a808197-05be-40f4-a97d-1f8ab741e991-kube-api-access-vhxz6\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:22.369617 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.369613    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a808197-05be-40f4-a97d-1f8ab741e991-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:22.369811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.369624    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a808197-05be-40f4-a97d-1f8ab741e991-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:22.369811 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.369633    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a808197-05be-40f4-a97d-1f8ab741e991-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:22.570334 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570259    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/1.log"
Sep 17 18:48:22.570676 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570662    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn_9a808197-05be-40f4-a97d-1f8ab741e991/storage-initializer/0.log"
Sep 17 18:48:22.570740 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570696    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a808197-05be-40f4-a97d-1f8ab741e991" containerID="f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5" exitCode=1
Sep 17 18:48:22.570800 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570782    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"
Sep 17 18:48:22.570840 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerDied","Data":"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"}
Sep 17 18:48:22.570897 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn" event={"ID":"9a808197-05be-40f4-a97d-1f8ab741e991","Type":"ContainerDied","Data":"4ae84dfcfe48562a23d3dcc37f1dd1f312270b3bcc4aa53b7fa8b0a5d9f7f892"}
Sep 17 18:48:22.570963 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.570952    2577 scope.go:117] "RemoveContainer" containerID="f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"
Sep 17 18:48:22.582127 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.582106    2577 scope.go:117] "RemoveContainer" containerID="ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"
Sep 17 18:48:22.589873 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.589855    2577 scope.go:117] "RemoveContainer" containerID="f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"
Sep 17 18:48:22.590134 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:48:22.590114    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5\": container with ID starting with f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5 not found: ID does not exist" containerID="f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"
Sep 17 18:48:22.590235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.590143    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5"} err="failed to get container status \"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5\": rpc error: code = NotFound desc = could not find container \"f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5\": container with ID starting with f7788198a004f1c615de2324e07dc5df7278e627330952c4544a0d543aee7ed5 not found: ID does not exist"
Sep 17 18:48:22.590235 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.590164    2577 scope.go:117] "RemoveContainer" containerID="ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"
Sep 17 18:48:22.590418 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:48:22.590394    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f\": container with ID starting with ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f not found: ID does not exist" containerID="ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"
Sep 17 18:48:22.590512 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.590422    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f"} err="failed to get container status \"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f\": rpc error: code = NotFound desc = could not find container \"ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f\": container with ID starting with ef286e77fa30ca71c50eab8ac43f730e1eca167132f521f7a8b9b39655ec2b5f not found: ID does not exist"
Sep 17 18:48:22.605864 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.605843    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:22.609181 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.609162    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-55c4455568-4rvrn"]
Sep 17 18:48:22.941782 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:22.941738    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" path="/var/lib/kubelet/pods/9a808197-05be-40f4-a97d-1f8ab741e991/volumes"
Sep 17 18:48:24.697825 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:24.697777    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:48:24.698240 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:24.698028    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" containerID="cri-o://f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833" gracePeriod=30
Sep 17 18:48:24.698240 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:24.698055    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kube-rbac-proxy" containerID="cri-o://ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c" gracePeriod=30
Sep 17 18:48:24.733644 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:24.733617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:25.589094 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:25.589056    2577 generic.go:356] "Generic (PLEG): container finished" podID="91eafde7-cda9-423a-b104-54c1024ad053" containerID="ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c" exitCode=2
Sep 17 18:48:25.589282 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:25.589140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerDied","Data":"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"}
Sep 17 18:48:27.335496 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.335460    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Sep 17 18:48:27.340246 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.340225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 17 18:48:27.543639 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.543617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:48:27.600692 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.600609    2577 generic.go:356] "Generic (PLEG): container finished" podID="91eafde7-cda9-423a-b104-54c1024ad053" containerID="f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833" exitCode=0
Sep 17 18:48:27.600692 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.600667    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"
Sep 17 18:48:27.600906 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.600683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerDied","Data":"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"}
Sep 17 18:48:27.600906 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.600791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7" event={"ID":"91eafde7-cda9-423a-b104-54c1024ad053","Type":"ContainerDied","Data":"04af65eee948e95198e8c893c43ddcfe89bcf01cf39ec127cb76536451e45eed"}
Sep 17 18:48:27.600906 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.600820    2577 scope.go:117] "RemoveContainer" containerID="ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"
Sep 17 18:48:27.609019 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.609005    2577 scope.go:117] "RemoveContainer" containerID="f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"
Sep 17 18:48:27.616186 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.616171    2577 scope.go:117] "RemoveContainer" containerID="16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"
Sep 17 18:48:27.623178 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623158    2577 scope.go:117] "RemoveContainer" containerID="ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"
Sep 17 18:48:27.623429 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:48:27.623411    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c\": container with ID starting with ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c not found: ID does not exist" containerID="ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"
Sep 17 18:48:27.623470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623437    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c"} err="failed to get container status \"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c\": rpc error: code = NotFound desc = could not find container \"ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c\": container with ID starting with ed2358e481c34e756c3d60e5598c656528c718e566226cc4e6e598a7a5882d9c not found: ID does not exist"
Sep 17 18:48:27.623470 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623454    2577 scope.go:117] "RemoveContainer" containerID="f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"
Sep 17 18:48:27.623691 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:48:27.623677    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833\": container with ID starting with f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833 not found: ID does not exist" containerID="f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"
Sep 17 18:48:27.623739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623694    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833"} err="failed to get container status \"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833\": rpc error: code = NotFound desc = could not find container \"f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833\": container with ID starting with f3293891fb74fa5432090eccc7b335a4d12f7362664b23404676b51840ded833 not found: ID does not exist"
Sep 17 18:48:27.623739 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623706    2577 scope.go:117] "RemoveContainer" containerID="16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"
Sep 17 18:48:27.623973 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:48:27.623955    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6\": container with ID starting with 16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6 not found: ID does not exist" containerID="16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"
Sep 17 18:48:27.624016 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.623984    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6"} err="failed to get container status \"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6\": rpc error: code = NotFound desc = could not find container \"16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6\": container with ID starting with 16228a6460088bd1ecd4770fb6f15107ee414c6fc29816d4ba0c6eeb9f8938c6 not found: ID does not exist"
Sep 17 18:48:27.715021 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.714991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s8f2\" (UniqueName: \"kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2\") pod \"91eafde7-cda9-423a-b104-54c1024ad053\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") "
Sep 17 18:48:27.715149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert\") pod \"91eafde7-cda9-423a-b104-54c1024ad053\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") "
Sep 17 18:48:27.715149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location\") pod \"91eafde7-cda9-423a-b104-54c1024ad053\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") "
Sep 17 18:48:27.715149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") pod \"91eafde7-cda9-423a-b104-54c1024ad053\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") "
Sep 17 18:48:27.715149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config\") pod \"91eafde7-cda9-423a-b104-54c1024ad053\" (UID: \"91eafde7-cda9-423a-b104-54c1024ad053\") "
Sep 17 18:48:27.715480 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91eafde7-cda9-423a-b104-54c1024ad053" (UID: "91eafde7-cda9-423a-b104-54c1024ad053"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:27.715560 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.715503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91eafde7-cda9-423a-b104-54c1024ad053" (UID: "91eafde7-cda9-423a-b104-54c1024ad053"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:27.717069 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.717036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91eafde7-cda9-423a-b104-54c1024ad053" (UID: "91eafde7-cda9-423a-b104-54c1024ad053"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:27.717241 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.717226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2" (OuterVolumeSpecName: "kube-api-access-6s8f2") pod "91eafde7-cda9-423a-b104-54c1024ad053" (UID: "91eafde7-cda9-423a-b104-54c1024ad053"). InnerVolumeSpecName "kube-api-access-6s8f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:27.729210 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.729186    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91eafde7-cda9-423a-b104-54c1024ad053" (UID: "91eafde7-cda9-423a-b104-54c1024ad053"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:48:27.816452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.816429    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8f2\" (UniqueName: \"kubernetes.io/projected/91eafde7-cda9-423a-b104-54c1024ad053-kube-api-access-6s8f2\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:27.816452 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.816449    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-cabundle-cert\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:27.816578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.816458    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eafde7-cda9-423a-b104-54c1024ad053-kserve-provision-location\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:27.816578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.816468    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eafde7-cda9-423a-b104-54c1024ad053-proxy-tls\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:27.816578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.816476    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eafde7-cda9-423a-b104-54c1024ad053-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:48:27.923843 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.923813    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:48:27.929715 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:27.929693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vknd7"]
Sep 17 18:48:28.942051 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:28.942018    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91eafde7-cda9-423a-b104-54c1024ad053" path="/var/lib/kubelet/pods/91eafde7-cda9-423a-b104-54c1024ad053/volumes"
Sep 17 18:48:29.858370 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:29.858344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:30.025814 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:30.025741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:57.499322 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:57.499281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:57.563153 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:48:57.563122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:13.934342 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:50:13.934268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:19.023347 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:50:19.023319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:19.353478 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:50:19.353385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:21.572910 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:50:21.572855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:11.823675 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:52:11.823645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:15.935429 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:52:15.935396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:16.224234 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:52:16.224160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:43.561681 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:52:43.561645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:01.138906 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:01.138872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:01.603149 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:01.603034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:19.213421 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:19.213390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:19.271390 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:19.271360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:25.065823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.065782    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ml4ch/must-gather-nftrt"]
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066264    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066283    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066304    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="storage-initializer"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066313    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="storage-initializer"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066325    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kube-rbac-proxy"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kube-rbac-proxy"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066345    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066354    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066367    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.066431 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066375    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.066977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066447    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.066977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kserve-container"
Sep 17 18:54:25.066977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eafde7-cda9-423a-b104-54c1024ad053" containerName="kube-rbac-proxy"
Sep 17 18:54:25.066977 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.066650    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a808197-05be-40f4-a97d-1f8ab741e991" containerName="storage-initializer"
Sep 17 18:54:25.069636 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.069615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.071827 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.071808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ml4ch\"/\"kube-root-ca.crt\""
Sep 17 18:54:25.071968 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.071951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ml4ch\"/\"openshift-service-ca.crt\""
Sep 17 18:54:25.078731 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.078706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ml4ch/must-gather-nftrt"]
Sep 17 18:54:25.081532 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.081506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:25.148717 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.148682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlpt\" (UniqueName: \"kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.148869 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.148735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.182365 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.182343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:25.249553 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.249527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlpt\" (UniqueName: \"kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.249666 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.249558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.249928 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.249911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.257853 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.257836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlpt\" (UniqueName: \"kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt\") pod \"must-gather-nftrt\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") " pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.378776 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.378733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:54:25.649087 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.649063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ml4ch/must-gather-nftrt"]
Sep 17 18:54:25.651213 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:54:25.651190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3335c2ab_e0e0_469e_9d45_f9327a9378a9.slice/crio-65c27b1be76010b30879c9845b49f30c5d26e670846e2312371ae879ff61d3a4 WatchSource:0}: Error finding container 65c27b1be76010b30879c9845b49f30c5d26e670846e2312371ae879ff61d3a4: Status 404 returned error can't find the container with id 65c27b1be76010b30879c9845b49f30c5d26e670846e2312371ae879ff61d3a4
Sep 17 18:54:25.652812 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:25.652791    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:54:26.131366 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:26.131325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ml4ch/must-gather-nftrt" event={"ID":"3335c2ab-e0e0-469e-9d45-f9327a9378a9","Type":"ContainerStarted","Data":"65c27b1be76010b30879c9845b49f30c5d26e670846e2312371ae879ff61d3a4"}
Sep 17 18:54:32.167848 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:32.167805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ml4ch/must-gather-nftrt" event={"ID":"3335c2ab-e0e0-469e-9d45-f9327a9378a9","Type":"ContainerStarted","Data":"15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46"}
Sep 17 18:54:32.168324 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:32.167859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ml4ch/must-gather-nftrt" event={"ID":"3335c2ab-e0e0-469e-9d45-f9327a9378a9","Type":"ContainerStarted","Data":"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"}
Sep 17 18:54:32.184311 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:32.184266    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ml4ch/must-gather-nftrt" podStartSLOduration=1.082641866 podStartE2EDuration="7.184253269s" totalImagesPullingTime="6.101611403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:32.18211128 +0000 UTC m=+3483.506432702" watchObservedRunningTime="2026-09-17 18:54:32.184253269 +0000 UTC m=+3483.508574693"
Sep 17 18:54:34.878090 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:54:34.878053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:00.304825 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:00.304791    2577 generic.go:356] "Generic (PLEG): container finished" podID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerID="ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138" exitCode=0
Sep 17 18:55:00.305315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:00.304823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ml4ch/must-gather-nftrt" event={"ID":"3335c2ab-e0e0-469e-9d45-f9327a9378a9","Type":"ContainerDied","Data":"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"}
Sep 17 18:55:00.305315 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:00.305103    2577 scope.go:117] "RemoveContainer" containerID="ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"
Sep 17 18:55:00.812433 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:00.812406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ml4ch_must-gather-nftrt_3335c2ab-e0e0-469e-9d45-f9327a9378a9/gather/0.log"
Sep 17 18:55:01.366947 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.366917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:01.489578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.489547    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mfgc/must-gather-6l9nb"]
Sep 17 18:55:01.493303 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.493285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.495692 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.495666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mfgc\"/\"openshift-service-ca.crt\""
Sep 17 18:55:01.495834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.495666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mfgc\"/\"kube-root-ca.crt\""
Sep 17 18:55:01.495834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.495769    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mfgc\"/\"default-dockercfg-mm674\""
Sep 17 18:55:01.502111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.502089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/must-gather-6l9nb"]
Sep 17 18:55:01.504230 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.504212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:01.564084 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.564033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62f53c3-c9a0-4140-8df2-4fcb43237815-must-gather-output\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.564268 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.564119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6wm\" (UniqueName: \"kubernetes.io/projected/f62f53c3-c9a0-4140-8df2-4fcb43237815-kube-api-access-cr6wm\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.566638 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.566613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:01.664736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.664642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62f53c3-c9a0-4140-8df2-4fcb43237815-must-gather-output\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.664736 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.664696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6wm\" (UniqueName: \"kubernetes.io/projected/f62f53c3-c9a0-4140-8df2-4fcb43237815-kube-api-access-cr6wm\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.665032 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.665012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62f53c3-c9a0-4140-8df2-4fcb43237815-must-gather-output\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.672525 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.672504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6wm\" (UniqueName: \"kubernetes.io/projected/f62f53c3-c9a0-4140-8df2-4fcb43237815-kube-api-access-cr6wm\") pod \"must-gather-6l9nb\" (UID: \"f62f53c3-c9a0-4140-8df2-4fcb43237815\") " pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:01.803222 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:01.803173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/must-gather-6l9nb"
Sep 17 18:55:02.073578 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:02.073553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/must-gather-6l9nb"]
Sep 17 18:55:02.075437 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:55:02.075412    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf62f53c3_c9a0_4140_8df2_4fcb43237815.slice/crio-9b396b20d83e40888824bea63c97d3eeb916c2cc61c5ef3ed3e6904c3983699c WatchSource:0}: Error finding container 9b396b20d83e40888824bea63c97d3eeb916c2cc61c5ef3ed3e6904c3983699c: Status 404 returned error can't find the container with id 9b396b20d83e40888824bea63c97d3eeb916c2cc61c5ef3ed3e6904c3983699c
Sep 17 18:55:02.314052 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:02.314015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/must-gather-6l9nb" event={"ID":"f62f53c3-c9a0-4140-8df2-4fcb43237815","Type":"ContainerStarted","Data":"9b396b20d83e40888824bea63c97d3eeb916c2cc61c5ef3ed3e6904c3983699c"}
Sep 17 18:55:03.322219 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:03.321696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/must-gather-6l9nb" event={"ID":"f62f53c3-c9a0-4140-8df2-4fcb43237815","Type":"ContainerStarted","Data":"b393bb9852de29c3a376c880d800f9547096758415b5bff1a359c14309010f63"}
Sep 17 18:55:03.322219 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:03.321773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/must-gather-6l9nb" event={"ID":"f62f53c3-c9a0-4140-8df2-4fcb43237815","Type":"ContainerStarted","Data":"1b4a76c3018f2c776d90071000d547383d65451e3a34418927849d6a07accf46"}
Sep 17 18:55:03.339055 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:03.338983    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mfgc/must-gather-6l9nb" podStartSLOduration=1.523629992 podStartE2EDuration="2.338957809s" totalImagesPullingTime="815.327817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:03.33700946 +0000 UTC m=+3514.661330882" watchObservedRunningTime="2026-09-17 18:55:03.338957809 +0000 UTC m=+3514.663279233"
Sep 17 18:55:04.506859 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:04.506829    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rkwq9_554048ca-bb0c-47aa-abbd-8c853cb84489/global-pull-secret-syncer/0.log"
Sep 17 18:55:04.621928 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:04.621897    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log"
Sep 17 18:55:04.761378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:04.761283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-151.ec2.internal_5e82e4d71f8fc63cce18d0028f27b2c2/haproxy/0.log"
Sep 17 18:55:06.349943 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.349898    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ml4ch/must-gather-nftrt"]
Sep 17 18:55:06.350707 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.350676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-ml4ch/must-gather-nftrt" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="copy" containerID="cri-o://15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46" gracePeriod=2
Sep 17 18:55:06.353310 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:55:06.353279    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nftrt\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ml4ch\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:55:06.354337 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.354320    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ml4ch/must-gather-nftrt"]
Sep 17 18:55:06.736783 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.735193    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ml4ch_must-gather-nftrt_3335c2ab-e0e0-469e-9d45-f9327a9378a9/copy/0.log"
Sep 17 18:55:06.742399 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.742076    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:55:06.747771 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:55:06.744582    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nftrt\" is forbidden: User \"system:node:ip-10-0-133-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ml4ch\": no relationship found between node 'ip-10-0-133-151.ec2.internal' and this object" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:55:06.820406 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.820359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output\") pod \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") "
Sep 17 18:55:06.820600 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.820417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xlpt\" (UniqueName: \"kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt\") pod \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\" (UID: \"3335c2ab-e0e0-469e-9d45-f9327a9378a9\") "
Sep 17 18:55:06.823407 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.823372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt" (OuterVolumeSpecName: "kube-api-access-5xlpt") pod "3335c2ab-e0e0-469e-9d45-f9327a9378a9" (UID: "3335c2ab-e0e0-469e-9d45-f9327a9378a9"). InnerVolumeSpecName "kube-api-access-5xlpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:06.823534 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.823469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3335c2ab-e0e0-469e-9d45-f9327a9378a9" (UID: "3335c2ab-e0e0-469e-9d45-f9327a9378a9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:06.922042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.921995    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3335c2ab-e0e0-469e-9d45-f9327a9378a9-must-gather-output\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:55:06.922042 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.922044    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlpt\" (UniqueName: \"kubernetes.io/projected/3335c2ab-e0e0-469e-9d45-f9327a9378a9-kube-api-access-5xlpt\") on node \"ip-10-0-133-151.ec2.internal\" DevicePath \"\""
Sep 17 18:55:06.946213 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:06.944923    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" path="/var/lib/kubelet/pods/3335c2ab-e0e0-469e-9d45-f9327a9378a9/volumes"
Sep 17 18:55:07.348804 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.348713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ml4ch_must-gather-nftrt_3335c2ab-e0e0-469e-9d45-f9327a9378a9/copy/0.log"
Sep 17 18:55:07.349438 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.349416    2577 generic.go:356] "Generic (PLEG): container finished" podID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerID="15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46" exitCode=143
Sep 17 18:55:07.349733 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.349721    2577 scope.go:117] "RemoveContainer" containerID="15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46"
Sep 17 18:55:07.350038 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.350026    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ml4ch/must-gather-nftrt"
Sep 17 18:55:07.358110 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.358083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:07.364782 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.364690    2577 scope.go:117] "RemoveContainer" containerID="ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"
Sep 17 18:55:07.384175 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.384156    2577 scope.go:117] "RemoveContainer" containerID="15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46"
Sep 17 18:55:07.384823 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:55:07.384676    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46\": container with ID starting with 15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46 not found: ID does not exist" containerID="15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46"
Sep 17 18:55:07.384823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.384710    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46"} err="failed to get container status \"15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46\": rpc error: code = NotFound desc = could not find container \"15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46\": container with ID starting with 15475c6fd4e5b823dc6e43edfdba844025d57fa7acefda1e095fbd9cd4199a46 not found: ID does not exist"
Sep 17 18:55:07.384823 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.384736    2577 scope.go:117] "RemoveContainer" containerID="ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"
Sep 17 18:55:07.385358 ip-10-0-133-151 kubenswrapper[2577]: E0917 18:55:07.385299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138\": container with ID starting with ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138 not found: ID does not exist" containerID="ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"
Sep 17 18:55:07.385358 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.385325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138"} err="failed to get container status \"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138\": rpc error: code = NotFound desc = could not find container \"ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138\": container with ID starting with ab7f03008a68ec6ef8da389f1658f37ca8c0d3ce8b045372aeedf712e2a80138 not found: ID does not exist"
Sep 17 18:55:07.417491 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:07.417456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hrb8m_c3045519-3c50-4afd-84ed-1d0a67387ef0/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:09.187013 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.186979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/alertmanager/0.log"
Sep 17 18:55:09.216465 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.216388    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/config-reloader/0.log"
Sep 17 18:55:09.241389 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.241359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/kube-rbac-proxy-web/0.log"
Sep 17 18:55:09.265079 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.265051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/kube-rbac-proxy/0.log"
Sep 17 18:55:09.292599 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.292568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/kube-rbac-proxy-metric/0.log"
Sep 17 18:55:09.327118 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.327048    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/prom-label-proxy/0.log"
Sep 17 18:55:09.353065 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.353032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0dffac96-c450-4e20-a67a-0839b65791c5/init-config-reloader/0.log"
Sep 17 18:55:09.512111 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.512034    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b9c9c9954-hxm49_0508cfb5-7aa5-4d89-9f15-62f3f15c30be/metrics-server/0.log"
Sep 17 18:55:09.665928 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.665892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbj9_d34fa22a-dfcd-4422-b4af-3bcfdaafea0d/node-exporter/0.log"
Sep 17 18:55:09.689834 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.689807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbj9_d34fa22a-dfcd-4422-b4af-3bcfdaafea0d/kube-rbac-proxy/0.log"
Sep 17 18:55:09.725360 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.725329    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbj9_d34fa22a-dfcd-4422-b4af-3bcfdaafea0d/init-textfile/0.log"
Sep 17 18:55:09.832471 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.832390    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7vhjx_7799ffba-3b3f-456a-ac76-5d632f0c65be/kube-rbac-proxy-main/0.log"
Sep 17 18:55:09.859103 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.859077    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7vhjx_7799ffba-3b3f-456a-ac76-5d632f0c65be/kube-rbac-proxy-self/0.log"
Sep 17 18:55:09.882892 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.882859    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7vhjx_7799ffba-3b3f-456a-ac76-5d632f0c65be/openshift-state-metrics/0.log"
Sep 17 18:55:09.926576 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.925462    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/prometheus/0.log"
Sep 17 18:55:09.945121 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.945089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/config-reloader/0.log"
Sep 17 18:55:09.970959 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.970935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/thanos-sidecar/0.log"
Sep 17 18:55:09.995370 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:09.995342    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/kube-rbac-proxy-web/0.log"
Sep 17 18:55:10.018766 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.018724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/kube-rbac-proxy/0.log"
Sep 17 18:55:10.043863 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.043835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/kube-rbac-proxy-thanos/0.log"
Sep 17 18:55:10.071935 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.071901    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76a372ad-f8e0-4000-9485-0ea78f875a54/init-config-reloader/0.log"
Sep 17 18:55:10.153275 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.153247    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-2mp6v_0ac1636a-899a-43e8-a78e-1d6e3263fbfb/prometheus-operator-admission-webhook/0.log"
Sep 17 18:55:10.187845 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.187807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cfd497cc4-tlzf7_b926fbaf-c4a2-46ed-95a2-64605bd84ac4/telemeter-client/0.log"
Sep 17 18:55:10.213534 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.213499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cfd497cc4-tlzf7_b926fbaf-c4a2-46ed-95a2-64605bd84ac4/reload/0.log"
Sep 17 18:55:10.243005 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.242969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cfd497cc4-tlzf7_b926fbaf-c4a2-46ed-95a2-64605bd84ac4/kube-rbac-proxy/0.log"
Sep 17 18:55:10.281005 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.280973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/thanos-query/0.log"
Sep 17 18:55:10.319215 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.319166    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/kube-rbac-proxy-web/0.log"
Sep 17 18:55:10.353802 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.353767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/kube-rbac-proxy/0.log"
Sep 17 18:55:10.388040 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.388006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/prom-label-proxy/0.log"
Sep 17 18:55:10.416545 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.416520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/kube-rbac-proxy-rules/0.log"
Sep 17 18:55:10.449949 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:10.449904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dd797f6c-h5sf9_960d49a8-0691-4460-80e5-0064307f92ca/kube-rbac-proxy-metrics/0.log"
Sep 17 18:55:11.657572 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:11.657543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-ngjg7_5c46e482-11b3-4e33-b53d-4894f0d70a44/networking-console-plugin/0.log"
Sep 17 18:55:13.019824 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.019788    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"]
Sep 17 18:55:13.020420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020296    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="gather"
Sep 17 18:55:13.020420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020321    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="gather"
Sep 17 18:55:13.020420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020343    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="copy"
Sep 17 18:55:13.020420 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020353    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="copy"
Sep 17 18:55:13.020651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020466    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="gather"
Sep 17 18:55:13.020651 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.020484    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3335c2ab-e0e0-469e-9d45-f9327a9378a9" containerName="copy"
Sep 17 18:55:13.025815 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.025788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.032251 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.032216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"]
Sep 17 18:55:13.106173 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.106129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-sys\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.106378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.106202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-lib-modules\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.106378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.106271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2b4\" (UniqueName: \"kubernetes.io/projected/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-kube-api-access-7d2b4\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.106378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.106301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-podres\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.106378 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.106338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-proc\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.206771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2b4\" (UniqueName: \"kubernetes.io/projected/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-kube-api-access-7d2b4\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.206771 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-podres\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-proc\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-sys\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-podres\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-proc\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-sys\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207035 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.206993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-lib-modules\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.207383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.207097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-lib-modules\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.215569 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.215543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2b4\" (UniqueName: \"kubernetes.io/projected/a1b39bc1-9304-464e-a7dc-7c19433c9fe5-kube-api-access-7d2b4\") pod \"perf-node-gather-daemonset-xdfq7\" (UID: \"a1b39bc1-9304-464e-a7dc-7c19433c9fe5\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.341104 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.341019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:13.638795 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:13.638562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"]
Sep 17 18:55:13.642314 ip-10-0-133-151 kubenswrapper[2577]: W0917 18:55:13.642280    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda1b39bc1_9304_464e_a7dc_7c19433c9fe5.slice/crio-4cd15d3176fa693450e9af086545b5fbddaa7764544201d49d619dff5af0f74a WatchSource:0}: Error finding container 4cd15d3176fa693450e9af086545b5fbddaa7764544201d49d619dff5af0f74a: Status 404 returned error can't find the container with id 4cd15d3176fa693450e9af086545b5fbddaa7764544201d49d619dff5af0f74a
Sep 17 18:55:14.236258 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.236229    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9f4z_c061ac52-ce91-447d-86da-66497b456a39/dns/0.log"
Sep 17 18:55:14.284075 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.284049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9f4z_c061ac52-ce91-447d-86da-66497b456a39/kube-rbac-proxy/0.log"
Sep 17 18:55:14.363800 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.363778    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxjw4_3ed57787-1808-4845-a93a-d2ddea2000cd/dns-node-resolver/0.log"
Sep 17 18:55:14.390901 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.390869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7" event={"ID":"a1b39bc1-9304-464e-a7dc-7c19433c9fe5","Type":"ContainerStarted","Data":"3fc0a8e551bb3bf71707c06b04a7805c56df8f104d728edb04214d795e379f3b"}
Sep 17 18:55:14.391057 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.390922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7" event={"ID":"a1b39bc1-9304-464e-a7dc-7c19433c9fe5","Type":"ContainerStarted","Data":"4cd15d3176fa693450e9af086545b5fbddaa7764544201d49d619dff5af0f74a"}
Sep 17 18:55:14.391057 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.390957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:14.904833 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:14.904806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4hcl_214c7453-5da4-4c58-af2f-e64a308f7679/node-ca/0.log"
Sep 17 18:55:16.174015 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:16.173986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xfvxq_5f3271f0-9520-4826-b061-ea071ecbdc7a/serve-healthcheck-canary/0.log"
Sep 17 18:55:16.606221 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:16.606202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwhms_ae9ec7ba-9975-4973-b711-50c7195153d5/kube-rbac-proxy/0.log"
Sep 17 18:55:16.628467 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:16.628436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwhms_ae9ec7ba-9975-4973-b711-50c7195153d5/exporter/0.log"
Sep 17 18:55:16.650116 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:16.650085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwhms_ae9ec7ba-9975-4973-b711-50c7195153d5/extractor/0.log"
Sep 17 18:55:20.085730 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.085702    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t29mz_ecec4b8b-3a67-4e58-8736-3476d86eb86a/s3-init/0.log"
Sep 17 18:55:20.137813 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.137779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t29mz_ecec4b8b-3a67-4e58-8736-3476d86eb86a/download-hf-model/0.log"
Sep 17 18:55:20.165154 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.165134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-pxvpr_abf3ada0-0809-40db-a496-9337fe588b13/s3-tls-init-custom/0.log"
Sep 17 18:55:20.181656 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.181634    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-pxvpr_abf3ada0-0809-40db-a496-9337fe588b13/download-hf-model/0.log"
Sep 17 18:55:20.238163 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.238145    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f9hjh_f4fc806e-4daf-4df7-9cb5-d5127aa607c4/s3-tls-init-serving/0.log"
Sep 17 18:55:20.257286 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.257263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f9hjh_f4fc806e-4daf-4df7-9cb5-d5127aa607c4/download-hf-model/0.log"
Sep 17 18:55:20.410063 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.410035    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7"
Sep 17 18:55:20.427262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:20.427210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-xdfq7" podStartSLOduration=7.427196526 podStartE2EDuration="7.427196526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:14.408079489 +0000 UTC m=+3525.732400913" watchObservedRunningTime="2026-09-17 18:55:20.427196526 +0000 UTC m=+3531.751517950"
Sep 17 18:55:24.749140 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:24.749058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-hnjtk_8188259b-edce-4811-84c9-66b13ee1293b/migrator/0.log"
Sep 17 18:55:24.770618 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:24.770530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-hnjtk_8188259b-edce-4811-84c9-66b13ee1293b/graceful-termination/0.log"
Sep 17 18:55:26.099879 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.099854    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:55:26.121912 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.121860    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/egress-router-binary-copy/0.log"
Sep 17 18:55:26.143000 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.142979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/cni-plugins/0.log"
Sep 17 18:55:26.192177 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.192151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/bond-cni-plugin/0.log"
Sep 17 18:55:26.208792 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.208739    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/routeoverride-cni/0.log"
Sep 17 18:55:26.229845 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.229824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/whereabouts-cni-bincopy/0.log"
Sep 17 18:55:26.252383 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.252361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s7bq8_42511188-d864-4c34-afc0-11ad0c80708d/whereabouts-cni/0.log"
Sep 17 18:55:26.447614 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.447579    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bkksm_6f7488df-ca4b-445f-b7f8-f738c2e59fed/kube-multus/0.log"
Sep 17 18:55:26.624980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.624952    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zwqgb_edac866d-e49b-4787-b73a-7b65d31c2946/network-metrics-daemon/0.log"
Sep 17 18:55:26.643182 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:26.643154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zwqgb_edac866d-e49b-4787-b73a-7b65d31c2946/kube-rbac-proxy/0.log"
Sep 17 18:55:27.596716 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.596693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/ovn-controller/0.log"
Sep 17 18:55:27.613980 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.613954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/ovn-acl-logging/0.log"
Sep 17 18:55:27.639262 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.639241    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/kube-rbac-proxy-node/0.log"
Sep 17 18:55:27.661089 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.661065    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:55:27.682379 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.682359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/northd/0.log"
Sep 17 18:55:27.704844 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.704819    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/nbdb/0.log"
Sep 17 18:55:27.726766 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.726726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/sbdb/0.log"
Sep 17 18:55:27.777641 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:27.777608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qmqtk_f7fd071e-9de8-4a2d-b721-d8f9142e7469/ovnkube-controller/0.log"
Sep 17 18:55:28.948715 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:28.948692    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-txn98_adf49094-f22f-46d6-9fa9-6b9589be542f/check-endpoints/0.log"
Sep 17 18:55:29.021682 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:29.021654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wnm46_c3a7b40f-58f6-45a0-afc0-c79d940e6b9f/network-check-target-container/0.log"
Sep 17 18:55:29.985685 ip-10-0-133-151 kubenswrapper[2577]: I0917 18:55:29.985648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-nxrr6_0fd2c703-5270-479a-9cc7-bcfe8ff8b99b/iptables-alerter/0.log"